2026-02-12T03:49:11.281 INFO:root:teuthology version: 0.0.1.dev31+g5f66ecfb3 2026-02-12T03:49:11.294 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-12T03:49:11.372 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-11_20:27:55-rados-wip-yuri3-testing-2026-02-09-1550-tentacle-distro-default-trial/45778 branch: wip-yuri3-testing-2026-02-09-1550-tentacle description: rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/many msgr/async-v2only objectstore/{bluestore/{alloc$/{btree} base mem$/{low} onode-segment$/{1M} write$/{v2/{compr$/{no$/{no}} v2}}}} rados supported-random-distro$/{centos_latest}} email: null first_in_suite: false flavor: default job_id: '45778' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-11_20:27:55-rados-wip-yuri3-testing-2026-02-09-1550-tentacle-distro-default-trial no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri3-testing-2026-02-09-1550-tentacle ceph: conf: global: mon client directed command retry: 5 mon client hunt interval max multiple: 2 mon election default strategy: 1 mon mgr beacon grace: 90 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 1000 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: btree bluestore block size: 96636764160 bluestore fsck on mount: true bluestore onode segment size: 1024K bluestore write v2: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd mclock skip benchmark: true osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) sha1: 1279775fef08a9169052c6dd6a10581353958aac ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 1279775fef08a9169052c6dd6a10581353958aac thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri3-testing-2026-02-09-1550-tentacle sha1: 1279775fef08a9169052c6dd6a10581353958aac owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 4009 sha1: 1279775fef08a9169052c6dd6a10581353958aac sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2026-02-09-1550-tentacle suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_1279775fef08a9169052c6dd6a10581353958aac/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 1279775fef08a9169052c6dd6a10581353958aac targets: trial103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIKSZYnefBHqnB+Zs4qYd7szXoh3dS315Y2eEMDVeUg71wl4BSpqk09Thxt77eUFfdHRY/KX8z744A/dV9l47ww= tasks: - install: null - ceph: conf: osd: osd max pg log entries: 5 osd min pg log entries: 5 create_rbd_pool: false log-ignorelist: - Error - overall HEALTH_ - \(OBJECT_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_BACKFILLFULL\) - \(POOL_NEARFULL\) - \(SLOW_OPS\) - \(TOO_FEW_PGS\) - Monitor daemon marked osd\.[[:digit:]]+ down, but it is still running - slow request - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force - backfill_toofull: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: reimage-ssh-down teuthology_sha1: 5f66ecfb34c0370410e78b3ee641753d19da653b timestamp: 2026-02-11_20:27:55 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.2161924 2026-02-12T03:49:11.372 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_1279775fef08a9169052c6dd6a10581353958aac/qa; will attempt to use it 2026-02-12T03:49:11.372 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_1279775fef08a9169052c6dd6a10581353958aac/qa/tasks 2026-02-12T03:49:11.372 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-12T03:49:11.373 INFO:teuthology.task.internal:Checking packages... 2026-02-12T03:49:11.374 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1279775fef08a9169052c6dd6a10581353958aac' 2026-02-12T03:49:11.374 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-12T03:49:11.374 INFO:teuthology.packaging:ref: None 2026-02-12T03:49:11.374 INFO:teuthology.packaging:tag: None 2026-02-12T03:49:11.374 INFO:teuthology.packaging:branch: wip-yuri3-testing-2026-02-09-1550-tentacle 2026-02-12T03:49:11.374 INFO:teuthology.packaging:sha1: 1279775fef08a9169052c6dd6a10581353958aac 2026-02-12T03:49:11.374 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri3-testing-2026-02-09-1550-tentacle 2026-02-12T03:49:11.508 INFO:teuthology.task.internal:Found packages for ceph version 20.2.0-626.g1279775f 2026-02-12T03:49:11.509 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-12T03:49:11.520 INFO:teuthology.task.internal:no buildpackages task found 2026-02-12T03:49:11.520 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-12T03:49:11.533 INFO:teuthology.task.internal:Saving configuration 2026-02-12T03:49:11.547 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-12T03:49:11.559 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-12T03:49:11.645 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial103.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-11_20:27:55-rados-wip-yuri3-testing-2026-02-09-1550-tentacle-distro-default-trial/45778', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-12 03:45:54.353967', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIKSZYnefBHqnB+Zs4qYd7szXoh3dS315Y2eEMDVeUg71wl4BSpqk09Thxt77eUFfdHRY/KX8z744A/dV9l47ww='} 2026-02-12T03:49:11.645 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-12T03:49:11.659 INFO:teuthology.task.internal:roles: ubuntu@trial103.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2026-02-12T03:49:11.659 INFO:teuthology.run_tasks:Running task console_log... 2026-02-12T03:49:11.746 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x76c34a7f3560>, signals=[15]) 2026-02-12T03:49:11.746 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-12T03:49:11.757 INFO:teuthology.task.internal:Opening connections... 2026-02-12T03:49:11.757 DEBUG:teuthology.task.internal:connecting to ubuntu@trial103.front.sepia.ceph.com 2026-02-12T03:49:11.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T03:49:11.822 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-12T03:49:11.833 DEBUG:teuthology.orchestra.run.trial103:> uname -m 2026-02-12T03:49:11.892 INFO:teuthology.orchestra.run.trial103.stdout:x86_64 2026-02-12T03:49:11.893 DEBUG:teuthology.orchestra.run.trial103:> cat /etc/os-release 2026-02-12T03:49:11.949 INFO:teuthology.orchestra.run.trial103.stdout:NAME="CentOS Stream" 2026-02-12T03:49:11.949 INFO:teuthology.orchestra.run.trial103.stdout:VERSION="9" 2026-02-12T03:49:11.949 INFO:teuthology.orchestra.run.trial103.stdout:ID="centos" 2026-02-12T03:49:11.949 INFO:teuthology.orchestra.run.trial103.stdout:ID_LIKE="rhel fedora" 2026-02-12T03:49:11.949 INFO:teuthology.orchestra.run.trial103.stdout:VERSION_ID="9" 2026-02-12T03:49:11.950 INFO:teuthology.orchestra.run.trial103.stdout:PLATFORM_ID="platform:el9" 2026-02-12T03:49:11.950 INFO:teuthology.orchestra.run.trial103.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-12T03:49:11.950 INFO:teuthology.orchestra.run.trial103.stdout:ANSI_COLOR="0;31" 2026-02-12T03:49:11.950 INFO:teuthology.orchestra.run.trial103.stdout:LOGO="fedora-logo-icon" 2026-02-12T03:49:11.950 INFO:teuthology.orchestra.run.trial103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-12T03:49:11.950 INFO:teuthology.orchestra.run.trial103.stdout:HOME_URL="https://centos.org/" 2026-02-12T03:49:11.950 INFO:teuthology.orchestra.run.trial103.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-12T03:49:11.950 INFO:teuthology.orchestra.run.trial103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-12T03:49:11.950 INFO:teuthology.orchestra.run.trial103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-12T03:49:11.951 INFO:teuthology.lock.ops:Updating trial103.front.sepia.ceph.com on lock server 2026-02-12T03:49:12.039 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-12T03:49:12.050 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-12T03:49:12.063 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-12T03:49:12.064 DEBUG:teuthology.orchestra.run.trial103:> test '!' -e /home/ubuntu/cephtest 2026-02-12T03:49:12.079 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-12T03:49:12.089 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-12T03:49:12.090 DEBUG:teuthology.orchestra.run.trial103:> test -z $(ls -A /var/lib/ceph) 2026-02-12T03:49:12.135 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-12T03:49:12.240 INFO:teuthology.run_tasks:Running task kernel... 2026-02-12T03:49:12.262 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-12T03:49:12.262 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-12T03:49:12.262 DEBUG:teuthology.orchestra.run.trial103:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-12T03:49:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T03:49:12.277 DEBUG:teuthology.orchestra.run.trial103:> uname -r 2026-02-12T03:49:12.333 INFO:teuthology.orchestra.run.trial103.stdout:5.14.0-645.el9.x86_64 2026-02-12T03:49:12.333 INFO:teuthology.task.kernel:Running kernel on trial103: 5.14.0-645.el9.x86_64 2026-02-12T03:49:12.333 DEBUG:teuthology.orchestra.run.trial103:> sudo yum install -y kernel 2026-02-12T03:49:20.763 INFO:teuthology.orchestra.run.trial103.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.9 MB 00:08 2026-02-12T03:49:23.941 INFO:teuthology.orchestra.run.trial103.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-12T03:49:27.318 INFO:teuthology.orchestra.run.trial103.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 7.8 MB 00:01 2026-02-12T03:49:28.333 INFO:teuthology.orchestra.run.trial103.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 20 kB 00:00 2026-02-12T03:49:29.549 INFO:teuthology.orchestra.run.trial103.stdout:Extra Packages for Enterprise Linux 18 MB/s | 20 MB 00:01 2026-02-12T03:49:32.721 INFO:teuthology.orchestra.run.trial103.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-12T03:49:33.785 INFO:teuthology.orchestra.run.trial103.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-12T03:49:33.806 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:49:33.807 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:49:33.807 INFO:teuthology.orchestra.run.trial103.stdout: Package Architecture Version Repository Size 2026-02-12T03:49:33.807 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:49:33.807 INFO:teuthology.orchestra.run.trial103.stdout:Installing: 2026-02-12T03:49:33.807 INFO:teuthology.orchestra.run.trial103.stdout: kernel x86_64 5.14.0-677.el9 baseos 768 k 2026-02-12T03:49:33.807 INFO:teuthology.orchestra.run.trial103.stdout:Installing dependencies: 2026-02-12T03:49:33.807 INFO:teuthology.orchestra.run.trial103.stdout: kernel-core x86_64 5.14.0-677.el9 baseos 17 M 2026-02-12T03:49:33.807 INFO:teuthology.orchestra.run.trial103.stdout: kernel-modules x86_64 5.14.0-677.el9 baseos 40 M 2026-02-12T03:49:33.807 INFO:teuthology.orchestra.run.trial103.stdout: kernel-modules-core x86_64 5.14.0-677.el9 baseos 31 M 2026-02-12T03:49:33.808 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:49:33.808 INFO:teuthology.orchestra.run.trial103.stdout:Transaction Summary 2026-02-12T03:49:33.808 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:49:33.808 INFO:teuthology.orchestra.run.trial103.stdout:Install 4 Packages 2026-02-12T03:49:33.808 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:49:33.808 INFO:teuthology.orchestra.run.trial103.stdout:Total download size: 88 M 2026-02-12T03:49:33.808 INFO:teuthology.orchestra.run.trial103.stdout:Installed size: 135 M 2026-02-12T03:49:33.808 INFO:teuthology.orchestra.run.trial103.stdout:Downloading Packages: 2026-02-12T03:49:34.759 INFO:teuthology.orchestra.run.trial103.stdout:(1/4): kernel-5.14.0-677.el9.x86_64.rpm 860 kB/s | 768 kB 00:00 2026-02-12T03:49:35.244 INFO:teuthology.orchestra.run.trial103.stdout:(2/4): kernel-core-5.14.0-677.el9.x86_64.rpm 12 MB/s | 17 MB 00:01 2026-02-12T03:49:35.708 INFO:teuthology.orchestra.run.trial103.stdout:(3/4): kernel-modules-5.14.0-677.el9.x86_64.rpm 22 MB/s | 40 MB 00:01 2026-02-12T03:49:36.455 INFO:teuthology.orchestra.run.trial103.stdout:(4/4): kernel-modules-core-5.14.0-677.el9.x86_6 18 MB/s | 31 MB 00:01 2026-02-12T03:49:36.455 INFO:teuthology.orchestra.run.trial103.stdout:-------------------------------------------------------------------------------- 2026-02-12T03:49:36.455 INFO:teuthology.orchestra.run.trial103.stdout:Total 33 MB/s | 88 MB 00:02 2026-02-12T03:49:36.645 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction check 2026-02-12T03:49:36.849 INFO:teuthology.orchestra.run.trial103.stdout:Transaction check succeeded. 2026-02-12T03:49:36.849 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction test 2026-02-12T03:49:37.194 INFO:teuthology.orchestra.run.trial103.stdout:Transaction test succeeded. 2026-02-12T03:49:37.194 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction 2026-02-12T03:49:38.284 INFO:teuthology.orchestra.run.trial103.stdout: Preparing : 1/1 2026-02-12T03:49:38.568 INFO:teuthology.orchestra.run.trial103.stdout: Installing : kernel-modules-core-5.14.0-677.el9.x86_64 1/4 2026-02-12T03:49:38.587 INFO:teuthology.orchestra.run.trial103.stdout: Installing : kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T03:49:40.353 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T03:49:40.382 INFO:teuthology.orchestra.run.trial103.stdout: Installing : kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T03:49:43.403 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T03:49:43.410 INFO:teuthology.orchestra.run.trial103.stdout: Installing : kernel-5.14.0-677.el9.x86_64 4/4 2026-02-12T03:49:46.448 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: kernel-modules-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T03:50:16.848 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: kernel-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T03:50:16.869 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: kernel-modules-5.14.0-677.el9.x86_64 4/4 2026-02-12T03:50:16.932 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: kernel-5.14.0-677.el9.x86_64 4/4 2026-02-12T03:50:16.932 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : kernel-5.14.0-677.el9.x86_64 1/4 2026-02-12T03:50:16.933 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T03:50:16.933 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T03:50:16.984 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : kernel-modules-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T03:50:16.984 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:50:16.984 INFO:teuthology.orchestra.run.trial103.stdout:Installed: 2026-02-12T03:50:16.984 INFO:teuthology.orchestra.run.trial103.stdout: kernel-5.14.0-677.el9.x86_64 2026-02-12T03:50:16.984 INFO:teuthology.orchestra.run.trial103.stdout: kernel-core-5.14.0-677.el9.x86_64 2026-02-12T03:50:16.984 INFO:teuthology.orchestra.run.trial103.stdout: kernel-modules-5.14.0-677.el9.x86_64 2026-02-12T03:50:16.984 INFO:teuthology.orchestra.run.trial103.stdout: kernel-modules-core-5.14.0-677.el9.x86_64 2026-02-12T03:50:16.984 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:50:16.985 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:50:17.042 DEBUG:teuthology.orchestra.run.trial103:> rpm -q kernel | sort -rV | head -n 1 2026-02-12T03:50:17.068 INFO:teuthology.orchestra.run.trial103.stdout:kernel-5.14.0-677.el9.x86_64 2026-02-12T03:50:17.069 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-677.el9.x86_64 2026-02-12T03:50:17.069 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-645.el9.x86_64 Expected: 5.14.0-677.el9.x86_64 2026-02-12T03:50:17.069 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-12T03:50:17.069 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-12T03:50:17.069 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-12T03:50:17.069 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial103.front.sepia.ceph.com, path=None, version=distro) 2026-02-12T03:50:17.070 DEBUG:teuthology.orchestra.run.trial103:> rpm -q kernel | sort -rV | head -n 1 2026-02-12T03:50:17.132 INFO:teuthology.orchestra.run.trial103.stdout:kernel-5.14.0-677.el9.x86_64 2026-02-12T03:50:17.133 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-677.el9.x86_64 2026-02-12T03:50:17.133 DEBUG:teuthology.orchestra.run.trial103:> sudo rpm -qi grub2-tools 2026-02-12T03:50:17.202 INFO:teuthology.orchestra.run.trial103.stdout:Name : grub2-tools 2026-02-12T03:50:17.202 INFO:teuthology.orchestra.run.trial103.stdout:Epoch : 1 2026-02-12T03:50:17.202 INFO:teuthology.orchestra.run.trial103.stdout:Version : 2.06 2026-02-12T03:50:17.202 INFO:teuthology.orchestra.run.trial103.stdout:Release : 118.el9 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:Architecture: x86_64 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:Install Date: Thu 04 Dec 2025 11:54:21 PM UTC 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:Group : Unspecified 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:Size : 8393909 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:License : GPLv3+ 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:Signature : RSA/SHA256, Wed 05 Nov 2025 09:13:15 AM UTC, Key ID 05b555b38483c65d 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:Source RPM : grub2-2.06-118.el9.src.rpm 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:Build Date : Tue 04 Nov 2025 03:44:14 PM UTC 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:Packager : builder@centos.org 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:Vendor : CentOS 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:Summary : Support tools for GRUB. 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:Description : 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:hardware devices. 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:50:17.203 INFO:teuthology.orchestra.run.trial103.stdout:This subpackage provides tools for support of all platforms. 2026-02-12T03:50:17.204 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-12T03:50:17.204 INFO:teuthology.task.kernel:Updating grub on trial103 to boot 5.14.0-677.el9.x86_64 2026-02-12T03:50:17.204 DEBUG:teuthology.orchestra.run.trial103:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-12T03:50:17.267 INFO:teuthology.orchestra.run.trial103.stdout:/boot/loader/entries/2734456005d743dca7f3dc77c35a2d0d-0-rescue.conf 2026-02-12T03:50:17.268 DEBUG:teuthology.orchestra.run.trial103:> sudo command -v grubby && echo yes || echo no 2026-02-12T03:50:17.331 INFO:teuthology.orchestra.run.trial103.stdout:/sbin/grubby 2026-02-12T03:50:17.332 INFO:teuthology.orchestra.run.trial103.stdout:yes 2026-02-12T03:50:17.333 DEBUG:teuthology.orchestra.run.trial103:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-677.el9.x86_64" | head -n 1 2026-02-12T03:50:17.396 INFO:teuthology.orchestra.run.trial103.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-677.el9.x86_64.conf 2026-02-12T03:50:17.397 DEBUG:teuthology.orchestra.run.trial103:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-677.el9.x86_64 2026-02-12T03:50:17.470 DEBUG:teuthology.orchestra.run.trial103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-12T03:50:17.679 INFO:teuthology.orchestra.run.trial103.stderr:Generating grub configuration file ... 2026-02-12T03:50:17.741 INFO:teuthology.orchestra.run.trial103.stderr:Found linux image: /boot/vmlinuz-5.14.0-677.el9.x86_64 2026-02-12T03:50:17.748 INFO:teuthology.orchestra.run.trial103.stderr:Found initrd image: /boot/initramfs-5.14.0-677.el9.x86_64.img 2026-02-12T03:50:17.997 INFO:teuthology.orchestra.run.trial103.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-12T03:50:18.003 INFO:teuthology.orchestra.run.trial103.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-12T03:50:18.034 INFO:teuthology.orchestra.run.trial103.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-12T03:50:18.041 INFO:teuthology.orchestra.run.trial103.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-12T03:50:18.065 INFO:teuthology.orchestra.run.trial103.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-12T03:50:18.071 INFO:teuthology.orchestra.run.trial103.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-12T03:50:18.155 INFO:teuthology.orchestra.run.trial103.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-12T03:50:18.163 INFO:teuthology.orchestra.run.trial103.stderr:done 2026-02-12T03:50:18.166 DEBUG:teuthology.orchestra.run.trial103:> test -d /sys/firmware/efi 2026-02-12T03:50:18.181 DEBUG:teuthology.orchestra.run.trial103:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-12T03:50:18.247 INFO:teuthology.orchestra.run.trial103.stdout:/boot/efi/EFI/BOOT 2026-02-12T03:50:18.247 INFO:teuthology.orchestra.run.trial103.stdout:/boot/efi/EFI/centos 2026-02-12T03:50:18.249 DEBUG:teuthology.orchestra.run.trial103:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-12T03:50:18.313 INFO:teuthology.orchestra.run.trial103.stdout:no 2026-02-12T03:50:18.313 DEBUG:teuthology.orchestra.run.trial103:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-12T03:50:18.377 INFO:teuthology.orchestra.run.trial103.stdout:yes 2026-02-12T03:50:18.378 DEBUG:teuthology.orchestra.run.trial103:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-12T03:50:18.442 DEBUG:teuthology.orchestra.run.trial103:> sudo shutdown -r now 2026-02-12T03:50:18.517 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2026-02-12T03:50:48.514 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-12T03:50:48.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial103.front.sepia.ceph.com' 2026-02-12T03:50:48.516 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T03:51:48.577 DEBUG:teuthology.orchestra.remote:timed out 2026-02-12T03:51:57.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial103.front.sepia.ceph.com' 2026-02-12T03:51:57.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T03:51:57.725 DEBUG:teuthology.orchestra.run.trial103:> true 2026-02-12T03:51:57.880 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial103.front.sepia.ceph.com' 2026-02-12T03:51:57.880 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-677.el9.x86_64"... 2026-02-12T03:51:57.880 DEBUG:teuthology.orchestra.run.trial103:> uname -r 2026-02-12T03:51:57.935 INFO:teuthology.orchestra.run.trial103.stdout:5.14.0-677.el9.x86_64 2026-02-12T03:51:57.936 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-677.el9.x86_64 vs 5.14.0-677.el9.x86_64 2026-02-12T03:51:57.936 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-12T03:51:57.936 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-12T03:51:58.937 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-12T03:51:58.937 DEBUG:teuthology.orchestra.run.trial103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-12T03:51:58.967 INFO:teuthology.orchestra.run.trial103.stdout:ttyS1 2026-02-12T03:51:58.992 DEBUG:teuthology.parallel:result is None 2026-02-12T03:51:58.993 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-12T03:51:59.004 INFO:teuthology.task.internal:Creating test directory... 2026-02-12T03:51:59.005 DEBUG:teuthology.orchestra.run.trial103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-12T03:51:59.050 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-12T03:51:59.062 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-12T03:51:59.076 INFO:teuthology.task.internal:Creating archive directory... 2026-02-12T03:51:59.076 DEBUG:teuthology.orchestra.run.trial103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-12T03:51:59.115 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-12T03:51:59.127 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-12T03:51:59.127 DEBUG:teuthology.orchestra.run.trial103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-12T03:51:59.172 INFO:teuthology.orchestra.run.trial103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-12T03:51:59.181 INFO:teuthology.orchestra.run.trial103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-12T03:51:59.183 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-12T03:51:59.194 INFO:teuthology.task.internal:Configuring sudo... 2026-02-12T03:51:59.195 DEBUG:teuthology.orchestra.run.trial103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-12T03:51:59.247 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-12T03:51:59.264 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-12T03:51:59.264 DEBUG:teuthology.orchestra.run.trial103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-12T03:51:59.304 DEBUG:teuthology.orchestra.run.trial103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T03:51:59.461 DEBUG:teuthology.orchestra.run.trial103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T03:51:59.487 DEBUG:teuthology.orchestra.run.trial103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T03:51:59.629 DEBUG:teuthology.orchestra.run.trial103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T03:51:59.653 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:51:59.653 DEBUG:teuthology.orchestra.run.trial103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-12T03:51:59.717 DEBUG:teuthology.orchestra.run.trial103:> sudo service rsyslog restart 2026-02-12T03:51:59.784 INFO:teuthology.orchestra.run.trial103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-12T03:51:59.989 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-12T03:52:00.001 INFO:teuthology.task.internal:Starting timer... 2026-02-12T03:52:00.002 INFO:teuthology.run_tasks:Running task pcp... 2026-02-12T03:52:00.017 INFO:teuthology.run_tasks:Running task selinux... 2026-02-12T03:52:00.030 DEBUG:teuthology.orchestra.run.trial103:> sudo service auditd rotate 2026-02-12T03:52:00.057 INFO:teuthology.orchestra.run.trial103.stdout:Rotating logs: 2026-02-12T03:52:00.059 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-12T03:52:00.059 DEBUG:teuthology.orchestra.run.trial103:> /usr/sbin/getenforce 2026-02-12T03:52:00.116 INFO:teuthology.orchestra.run.trial103.stdout:Permissive 2026-02-12T03:52:00.116 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial103.front.sepia.ceph.com': 'permissive'} 2026-02-12T03:52:00.116 DEBUG:teuthology.orchestra.run.trial103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-12T03:52:00.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T03:52:00.181 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-12T03:52:00.181 DEBUG:teuthology.orchestra.run.trial103:> sudo /usr/sbin/setenforce permissive 2026-02-12T03:52:00.244 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-12T03:52:00.257 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-12T03:52:00.264 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-12T03:52:00.264 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial103.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-12T03:53:08.294 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial103.front.sepia.ceph.com')] 2026-02-12T03:53:08.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial103.front.sepia.ceph.com' 2026-02-12T03:53:08.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T03:53:08.411 DEBUG:teuthology.orchestra.run.trial103:> true 2026-02-12T03:53:08.491 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial103.front.sepia.ceph.com' 2026-02-12T03:53:08.491 INFO:teuthology.run_tasks:Running task clock... 2026-02-12T03:53:08.505 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-12T03:53:08.505 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-12T03:53:08.506 DEBUG:teuthology.orchestra.run.trial103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-12T03:53:08.559 INFO:teuthology.orchestra.run.trial103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-12T03:53:08.571 INFO:teuthology.orchestra.run.trial103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-12T03:53:08.591 INFO:teuthology.orchestra.run.trial103.stderr:sudo: ntpd: command not found 2026-02-12T03:53:08.601 INFO:teuthology.orchestra.run.trial103.stdout:506 Cannot talk to daemon 2026-02-12T03:53:08.613 INFO:teuthology.orchestra.run.trial103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-12T03:53:08.624 INFO:teuthology.orchestra.run.trial103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-12T03:53:08.670 INFO:teuthology.orchestra.run.trial103.stderr:bash: line 1: ntpq: command not found 2026-02-12T03:53:08.690 INFO:teuthology.orchestra.run.trial103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-12T03:53:08.690 INFO:teuthology.orchestra.run.trial103.stdout:=============================================================================== 2026-02-12T03:53:08.690 INFO:teuthology.orchestra.run.trial103.stdout:^? 170-187-147-56.ip.linode> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T03:53:08.690 INFO:teuthology.orchestra.run.trial103.stdout:^? us-central-1.clearnet.pw 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T03:53:08.690 INFO:teuthology.orchestra.run.trial103.stdout:^? time.cloudflare.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T03:53:08.690 INFO:teuthology.orchestra.run.trial103.stdout:^? time1.weber.edu 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T03:53:08.691 INFO:teuthology.run_tasks:Running task install... 2026-02-12T03:53:08.705 DEBUG:teuthology.task.install:project ceph 2026-02-12T03:53:08.705 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1279775fef08a9169052c6dd6a10581353958aac'}} 2026-02-12T03:53:08.705 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1279775fef08a9169052c6dd6a10581353958aac'} 2026-02-12T03:53:08.705 INFO:teuthology.task.install:Using flavor: default 2026-02-12T03:53:08.714 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-12T03:53:08.714 INFO:teuthology.task.install:extra packages: [] 2026-02-12T03:53:08.715 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1279775fef08a9169052c6dd6a10581353958aac', 'tag': None, 'wait_for_package': False} 2026-02-12T03:53:08.716 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1279775fef08a9169052c6dd6a10581353958aac 2026-02-12T03:53:08.898 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-09-1550-tentacle/1279775fef08a9169052c6dd6a10581353958aac/centos/9/flavors/default/ 2026-02-12T03:53:08.898 INFO:teuthology.task.install.rpm:Package version is 20.2.0-626.g1279775f 2026-02-12T03:53:09.239 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-09-1550-tentacle/1279775fef08a9169052c6dd6a10581353958aac/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-09-1550-tentacle/1279775fef08a9169052c6dd6a10581353958aac/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-09-1550-tentacle/1279775fef08a9169052c6dd6a10581353958aac/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-12T03:53:09.239 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:53:09.240 DEBUG:teuthology.orchestra.run.trial103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-12T03:53:09.267 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-12T03:53:09.267 DEBUG:teuthology.orchestra.run.trial103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1279775fef08a9169052c6dd6a10581353958aac/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-12T03:53:09.332 DEBUG:teuthology.orchestra.run.trial103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-12T03:53:09.406 DEBUG:teuthology.orchestra.run.trial103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-12T03:53:09.470 INFO:teuthology.orchestra.run.trial103.stdout:check_obsoletes = 1 2026-02-12T03:53:09.471 DEBUG:teuthology.orchestra.run.trial103:> sudo yum clean all 2026-02-12T03:53:09.619 INFO:teuthology.orchestra.run.trial103.stdout:41 files removed 2026-02-12T03:53:09.633 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install ceph-radosgw 2026-02-12T03:53:10.386 INFO:teuthology.orchestra.run.trial103.stdout:ceph packages for x86_64 146 kB/s | 89 kB 00:00 2026-02-12T03:53:10.980 INFO:teuthology.orchestra.run.trial103.stdout:ceph noarch packages 33 kB/s | 19 kB 00:00 2026-02-12T03:53:11.490 INFO:teuthology.orchestra.run.trial103.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2026-02-12T03:53:12.313 INFO:teuthology.orchestra.run.trial103.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-12T03:53:15.454 INFO:teuthology.orchestra.run.trial103.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-12T03:53:27.141 INFO:teuthology.orchestra.run.trial103.stdout:CentOS Stream 9 - CRB 864 kB/s | 7.8 MB 00:09 2026-02-12T03:53:28.200 INFO:teuthology.orchestra.run.trial103.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 20 kB 00:00 2026-02-12T03:53:28.487 INFO:teuthology.orchestra.run.trial103.stdout:Extra Packages for Enterprise Linux 96 MB/s | 20 MB 00:00 2026-02-12T03:53:31.729 INFO:teuthology.orchestra.run.trial103.stdout:lab-extras 1.2 MB/s | 50 kB 00:00 2026-02-12T03:53:33.005 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout: Package Arch Version Repository Size 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout:Installing: 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout: ceph-radosgw x86_64 2:20.2.0-626.g1279775f.el9 ceph 24 M 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout:Upgrading: 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout: librados2 x86_64 2:20.2.0-626.g1279775f.el9 ceph 3.5 M 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout: librbd1 x86_64 2:20.2.0-626.g1279775f.el9 ceph 2.8 M 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout:Installing dependencies: 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout: ceph-base x86_64 2:20.2.0-626.g1279775f.el9 ceph 5.9 M 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout: ceph-common x86_64 2:20.2.0-626.g1279775f.el9 ceph 24 M 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout: ceph-selinux x86_64 2:20.2.0-626.g1279775f.el9 ceph 25 k 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-12T03:53:33.007 INFO:teuthology.orchestra.run.trial103.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: libcephfs2 x86_64 2:20.2.0-626.g1279775f.el9 ceph 869 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: libradosstriper1 x86_64 2:20.2.0-626.g1279775f.el9 ceph 250 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: librgw2 x86_64 2:20.2.0-626.g1279775f.el9 ceph 6.4 M 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: python3-ceph-argparse x86_64 2:20.2.0-626.g1279775f.el9 ceph 45 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: python3-ceph-common x86_64 2:20.2.0-626.g1279775f.el9 ceph 175 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: python3-cephfs x86_64 2:20.2.0-626.g1279775f.el9 ceph 163 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: python3-rados x86_64 2:20.2.0-626.g1279775f.el9 ceph 323 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: python3-rbd x86_64 2:20.2.0-626.g1279775f.el9 ceph 304 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: python3-rgw x86_64 2:20.2.0-626.g1279775f.el9 ceph 99 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout:Installing weak dependencies: 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: libcephfs-daemon x86_64 2:20.2.0-626.g1279775f.el9 ceph 35 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: libcephfs-proxy2 x86_64 2:20.2.0-626.g1279775f.el9 ceph 24 k 2026-02-12T03:53:33.008 INFO:teuthology.orchestra.run.trial103.stdout: nvme-cli x86_64 2.13-1.el9 baseos 1.1 M 2026-02-12T03:53:33.009 INFO:teuthology.orchestra.run.trial103.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-12T03:53:33.009 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:33.009 INFO:teuthology.orchestra.run.trial103.stdout:Transaction Summary 2026-02-12T03:53:33.009 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:53:33.009 INFO:teuthology.orchestra.run.trial103.stdout:Install 35 Packages 2026-02-12T03:53:33.009 INFO:teuthology.orchestra.run.trial103.stdout:Upgrade 2 Packages 2026-02-12T03:53:33.009 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:33.009 INFO:teuthology.orchestra.run.trial103.stdout:Total download size: 79 M 2026-02-12T03:53:33.009 INFO:teuthology.orchestra.run.trial103.stdout:Downloading Packages: 2026-02-12T03:53:33.999 INFO:teuthology.orchestra.run.trial103.stdout:(1/37): ceph-base-20.2.0-626.g1279775f.el9.x86_ 12 MB/s | 5.9 MB 00:00 2026-02-12T03:53:34.024 INFO:teuthology.orchestra.run.trial103.stdout:(2/37): ceph-selinux-20.2.0-626.g1279775f.el9.x 1.0 MB/s | 25 kB 00:00 2026-02-12T03:53:34.049 INFO:teuthology.orchestra.run.trial103.stdout:(3/37): libcephfs-daemon-20.2.0-626.g1279775f.e 1.4 MB/s | 35 kB 00:00 2026-02-12T03:53:34.075 INFO:teuthology.orchestra.run.trial103.stdout:(4/37): libcephfs-proxy2-20.2.0-626.g1279775f.e 924 kB/s | 24 kB 00:00 2026-02-12T03:53:34.146 INFO:teuthology.orchestra.run.trial103.stdout:(5/37): libcephfs2-20.2.0-626.g1279775f.el9.x86 12 MB/s | 869 kB 00:00 2026-02-12T03:53:34.182 INFO:teuthology.orchestra.run.trial103.stdout:(6/37): libradosstriper1-20.2.0-626.g1279775f.e 6.9 MB/s | 250 kB 00:00 2026-02-12T03:53:34.619 INFO:teuthology.orchestra.run.trial103.stdout:(7/37): librgw2-20.2.0-626.g1279775f.el9.x86_64 15 MB/s | 6.4 MB 00:00 2026-02-12T03:53:34.658 INFO:teuthology.orchestra.run.trial103.stdout:(8/37): python3-ceph-argparse-20.2.0-626.g12797 1.1 MB/s | 45 kB 00:00 2026-02-12T03:53:34.704 INFO:teuthology.orchestra.run.trial103.stdout:(9/37): python3-ceph-common-20.2.0-626.g1279775 3.7 MB/s | 175 kB 00:00 2026-02-12T03:53:34.752 INFO:teuthology.orchestra.run.trial103.stdout:(10/37): python3-cephfs-20.2.0-626.g1279775f.el 3.3 MB/s | 163 kB 00:00 2026-02-12T03:53:34.805 INFO:teuthology.orchestra.run.trial103.stdout:(11/37): python3-rados-20.2.0-626.g1279775f.el9 6.0 MB/s | 323 kB 00:00 2026-02-12T03:53:34.859 INFO:teuthology.orchestra.run.trial103.stdout:(12/37): python3-rbd-20.2.0-626.g1279775f.el9.x 5.5 MB/s | 304 kB 00:00 2026-02-12T03:53:34.910 INFO:teuthology.orchestra.run.trial103.stdout:(13/37): python3-rgw-20.2.0-626.g1279775f.el9.x 1.9 MB/s | 99 kB 00:00 2026-02-12T03:53:35.183 INFO:teuthology.orchestra.run.trial103.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 122 kB/s | 33 kB 00:00 2026-02-12T03:53:35.372 INFO:teuthology.orchestra.run.trial103.stdout:(15/37): ceph-radosgw-20.2.0-626.g1279775f.el9. 13 MB/s | 24 MB 00:01 2026-02-12T03:53:35.541 INFO:teuthology.orchestra.run.trial103.stdout:(16/37): ceph-common-20.2.0-626.g1279775f.el9.x 12 MB/s | 24 MB 00:02 2026-02-12T03:53:35.667 INFO:teuthology.orchestra.run.trial103.stdout:(17/37): nvme-cli-2.13-1.el9.x86_64.rpm 2.3 MB/s | 1.1 MB 00:00 2026-02-12T03:53:35.707 INFO:teuthology.orchestra.run.trial103.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 4.0 MB/s | 164 kB 00:00 2026-02-12T03:53:35.734 INFO:teuthology.orchestra.run.trial103.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 5.8 MB/s | 160 kB 00:00 2026-02-12T03:53:35.758 INFO:teuthology.orchestra.run.trial103.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.8 MB/s | 45 kB 00:00 2026-02-12T03:53:35.761 INFO:teuthology.orchestra.run.trial103.stdout:(21/37): pciutils-3.7.0-7.el9.x86_64.rpm 239 kB/s | 93 kB 00:00 2026-02-12T03:53:35.767 INFO:teuthology.orchestra.run.trial103.stdout:(22/37): boost-program-options-1.75.0-13.el9.x8 458 kB/s | 104 kB 00:00 2026-02-12T03:53:35.791 INFO:teuthology.orchestra.run.trial103.stdout:(23/37): librdkafka-1.6.1-102.el9.x86_64.rpm 20 MB/s | 662 kB 00:00 2026-02-12T03:53:35.793 INFO:teuthology.orchestra.run.trial103.stdout:(24/37): qatlib-25.08.0-2.el9.x86_64.rpm 9.3 MB/s | 240 kB 00:00 2026-02-12T03:53:35.814 INFO:teuthology.orchestra.run.trial103.stdout:(25/37): qatlib-service-25.08.0-2.el9.x86_64.rp 1.6 MB/s | 37 kB 00:00 2026-02-12T03:53:35.818 INFO:teuthology.orchestra.run.trial103.stdout:(26/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 2.6 MB/s | 66 kB 00:00 2026-02-12T03:53:35.918 INFO:teuthology.orchestra.run.trial103.stdout:(27/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 292 kB 00:00 2026-02-12T03:53:36.108 INFO:teuthology.orchestra.run.trial103.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 130 kB/s | 25 kB 00:00 2026-02-12T03:53:36.154 INFO:teuthology.orchestra.run.trial103.stdout:(29/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 906 kB/s | 308 kB 00:00 2026-02-12T03:53:36.194 INFO:teuthology.orchestra.run.trial103.stdout:(30/37): liboath-2.6.12-1.el9.x86_64.rpm 577 kB/s | 49 kB 00:00 2026-02-12T03:53:36.202 INFO:teuthology.orchestra.run.trial103.stdout:(31/37): libunwind-1.6.2-1.el9.x86_64.rpm 1.4 MB/s | 67 kB 00:00 2026-02-12T03:53:36.261 INFO:teuthology.orchestra.run.trial103.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 3.1 MB/s | 191 kB 00:00 2026-02-12T03:53:36.333 INFO:teuthology.orchestra.run.trial103.stdout:(33/37): libarrow-9.0.0-15.el9.x86_64.rpm 8.6 MB/s | 4.4 MB 00:00 2026-02-12T03:53:36.397 INFO:teuthology.orchestra.run.trial103.stdout:(34/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 4.0 MB/s | 838 kB 00:00 2026-02-12T03:53:36.416 INFO:teuthology.orchestra.run.trial103.stdout:(35/37): thrift-0.15.0-4.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2026-02-12T03:53:36.449 INFO:teuthology.orchestra.run.trial103.stdout:(36/37): librados2-20.2.0-626.g1279775f.el9.x86 30 MB/s | 3.5 MB 00:00 2026-02-12T03:53:38.149 INFO:teuthology.orchestra.run.trial103.stdout:(37/37): librbd1-20.2.0-626.g1279775f.el9.x86_6 1.6 MB/s | 2.8 MB 00:01 2026-02-12T03:53:38.152 INFO:teuthology.orchestra.run.trial103.stdout:-------------------------------------------------------------------------------- 2026-02-12T03:53:38.152 INFO:teuthology.orchestra.run.trial103.stdout:Total 15 MB/s | 79 MB 00:05 2026-02-12T03:53:38.246 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction check 2026-02-12T03:53:38.265 INFO:teuthology.orchestra.run.trial103.stdout:Transaction check succeeded. 2026-02-12T03:53:38.265 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction test 2026-02-12T03:53:38.566 INFO:teuthology.orchestra.run.trial103.stdout:Transaction test succeeded. 2026-02-12T03:53:38.566 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction 2026-02-12T03:53:38.910 INFO:teuthology.orchestra.run.trial103.stdout: Preparing : 1/1 2026-02-12T03:53:38.921 INFO:teuthology.orchestra.run.trial103.stdout: Installing : thrift-0.15.0-4.el9.x86_64 1/39 2026-02-12T03:53:39.074 INFO:teuthology.orchestra.run.trial103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 2/39 2026-02-12T03:53:39.078 INFO:teuthology.orchestra.run.trial103.stdout: Upgrading : librados2-2:20.2.0-626.g1279775f.el9.x86_64 3/39 2026-02-12T03:53:39.125 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: librados2-2:20.2.0-626.g1279775f.el9.x86_64 3/39 2026-02-12T03:53:39.126 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libcephfs2-2:20.2.0-626.g1279775f.el9.x86_64 4/39 2026-02-12T03:53:39.138 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: libcephfs2-2:20.2.0-626.g1279775f.el9.x86_64 4/39 2026-02-12T03:53:39.140 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libcephfs-daemon-2:20.2.0-626.g1279775f.el9.x86_64 5/39 2026-02-12T03:53:39.141 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libcephfs-proxy2-2:20.2.0-626.g1279775f.el9.x86_64 6/39 2026-02-12T03:53:39.168 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: libcephfs-proxy2-2:20.2.0-626.g1279775f.el9.x86_64 6/39 2026-02-12T03:53:39.170 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-rados-2:20.2.0-626.g1279775f.el9.x86_64 7/39 2026-02-12T03:53:39.178 INFO:teuthology.orchestra.run.trial103.stdout: Installing : liboath-2.6.12-1.el9.x86_64 8/39 2026-02-12T03:53:39.180 INFO:teuthology.orchestra.run.trial103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 9/39 2026-02-12T03:53:39.184 INFO:teuthology.orchestra.run.trial103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 10/39 2026-02-12T03:53:39.188 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 11/39 2026-02-12T03:53:39.196 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-ceph-argparse-2:20.2.0-626.g1279775f.el9.x 12/39 2026-02-12T03:53:39.208 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-cephfs-2:20.2.0-626.g1279775f.el9.x86_64 13/39 2026-02-12T03:53:39.209 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libradosstriper1-2:20.2.0-626.g1279775f.el9.x86_64 14/39 2026-02-12T03:53:39.223 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: libradosstriper1-2:20.2.0-626.g1279775f.el9.x86_64 14/39 2026-02-12T03:53:39.255 INFO:teuthology.orchestra.run.trial103.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-12T03:53:39.259 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-12T03:53:39.265 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-12T03:53:39.267 INFO:teuthology.orchestra.run.trial103.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-12T03:53:39.272 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-12T03:53:39.394 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-12T03:53:39.395 INFO:teuthology.orchestra.run.trial103.stdout: Upgrading : librbd1-2:20.2.0-626.g1279775f.el9.x86_64 21/39 2026-02-12T03:53:39.421 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: librbd1-2:20.2.0-626.g1279775f.el9.x86_64 21/39 2026-02-12T03:53:39.423 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-rbd-2:20.2.0-626.g1279775f.el9.x86_64 22/39 2026-02-12T03:53:39.429 INFO:teuthology.orchestra.run.trial103.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-12T03:53:39.709 INFO:teuthology.orchestra.run.trial103.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-12T03:53:39.710 INFO:teuthology.orchestra.run.trial103.stdout: Installing : librgw2-2:20.2.0-626.g1279775f.el9.x86_64 25/39 2026-02-12T03:53:39.725 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: librgw2-2:20.2.0-626.g1279775f.el9.x86_64 25/39 2026-02-12T03:53:39.729 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-rgw-2:20.2.0-626.g1279775f.el9.x86_64 26/39 2026-02-12T03:53:39.734 INFO:teuthology.orchestra.run.trial103.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-12T03:53:39.737 INFO:teuthology.orchestra.run.trial103.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-12T03:53:39.738 INFO:teuthology.orchestra.run.trial103.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-12T03:53:39.755 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-12T03:53:39.868 INFO:teuthology.orchestra.run.trial103.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-12T03:53:39.871 INFO:teuthology.orchestra.run.trial103.stdout: Installing : nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-12T03:53:40.141 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-12T03:53:40.141 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-12T03:53:40.141 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:40.183 INFO:teuthology.orchestra.run.trial103.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-12T03:53:40.185 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-ceph-common-2:20.2.0-626.g1279775f.el9.x86 33/39 2026-02-12T03:53:41.264 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-common-2:20.2.0-626.g1279775f.el9.x86_64 34/39 2026-02-12T03:53:41.267 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-common-2:20.2.0-626.g1279775f.el9.x86_64 34/39 2026-02-12T03:53:41.561 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-common-2:20.2.0-626.g1279775f.el9.x86_64 34/39 2026-02-12T03:53:41.564 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-base-2:20.2.0-626.g1279775f.el9.x86_64 35/39 2026-02-12T03:53:41.605 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-base-2:20.2.0-626.g1279775f.el9.x86_64 35/39 2026-02-12T03:53:41.605 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-12T03:53:41.605 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-12T03:53:41.605 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:41.608 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-selinux-2:20.2.0-626.g1279775f.el9.x86_64 36/39 2026-02-12T03:53:46.129 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-selinux-2:20.2.0-626.g1279775f.el9.x86_64 36/39 2026-02-12T03:53:46.129 INFO:teuthology.orchestra.run.trial103.stdout:skipping the directory /sys 2026-02-12T03:53:46.129 INFO:teuthology.orchestra.run.trial103.stdout:skipping the directory /proc 2026-02-12T03:53:46.129 INFO:teuthology.orchestra.run.trial103.stdout:skipping the directory /mnt 2026-02-12T03:53:46.129 INFO:teuthology.orchestra.run.trial103.stdout:skipping the directory /var/tmp 2026-02-12T03:53:46.129 INFO:teuthology.orchestra.run.trial103.stdout:skipping the directory /home 2026-02-12T03:53:46.129 INFO:teuthology.orchestra.run.trial103.stdout:skipping the directory /root 2026-02-12T03:53:46.129 INFO:teuthology.orchestra.run.trial103.stdout:skipping the directory /tmp 2026-02-12T03:53:46.129 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:47.156 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-radosgw-2:20.2.0-626.g1279775f.el9.x86_64 37/39 2026-02-12T03:53:47.179 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-radosgw-2:20.2.0-626.g1279775f.el9.x86_64 37/39 2026-02-12T03:53:47.179 INFO:teuthology.orchestra.run.trial103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T03:53:47.179 INFO:teuthology.orchestra.run.trial103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-12T03:53:47.179 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-12T03:53:47.179 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-12T03:53:47.179 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:47.181 INFO:teuthology.orchestra.run.trial103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-12T03:53:47.194 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-12T03:53:47.194 INFO:teuthology.orchestra.run.trial103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T03:53:47.247 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T03:53:47.247 INFO:teuthology.orchestra.run.trial103.stdout:Creating group 'qat' with GID 992. 2026-02-12T03:53:47.247 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:47.545 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-base-2:20.2.0-626.g1279775f.el9.x86_64 1/39 2026-02-12T03:53:47.545 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-common-2:20.2.0-626.g1279775f.el9.x86_64 2/39 2026-02-12T03:53:47.545 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-radosgw-2:20.2.0-626.g1279775f.el9.x86_64 3/39 2026-02-12T03:53:47.545 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-selinux-2:20.2.0-626.g1279775f.el9.x86_64 4/39 2026-02-12T03:53:47.545 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libcephfs-daemon-2:20.2.0-626.g1279775f.el9.x86_64 5/39 2026-02-12T03:53:47.546 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libcephfs-proxy2-2:20.2.0-626.g1279775f.el9.x86_64 6/39 2026-02-12T03:53:47.546 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libcephfs2-2:20.2.0-626.g1279775f.el9.x86_64 7/39 2026-02-12T03:53:47.546 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libradosstriper1-2:20.2.0-626.g1279775f.el9.x86_64 8/39 2026-02-12T03:53:47.546 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : librgw2-2:20.2.0-626.g1279775f.el9.x86_64 9/39 2026-02-12T03:53:47.546 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-ceph-argparse-2:20.2.0-626.g1279775f.el9.x 10/39 2026-02-12T03:53:47.546 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-ceph-common-2:20.2.0-626.g1279775f.el9.x86 11/39 2026-02-12T03:53:47.546 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-cephfs-2:20.2.0-626.g1279775f.el9.x86_64 12/39 2026-02-12T03:53:47.546 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-rados-2:20.2.0-626.g1279775f.el9.x86_64 13/39 2026-02-12T03:53:47.546 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-rbd-2:20.2.0-626.g1279775f.el9.x86_64 14/39 2026-02-12T03:53:47.546 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-rgw-2:20.2.0-626.g1279775f.el9.x86_64 15/39 2026-02-12T03:53:47.547 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-12T03:53:47.547 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : nvme-cli-2.13-1.el9.x86_64 17/39 2026-02-12T03:53:47.547 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-12T03:53:47.547 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-12T03:53:47.547 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-12T03:53:47.547 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-12T03:53:47.547 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-12T03:53:47.547 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-12T03:53:47.547 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-12T03:53:47.547 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-12T03:53:47.547 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-12T03:53:47.548 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-12T03:53:47.548 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-12T03:53:47.548 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-12T03:53:47.548 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-12T03:53:47.548 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-12T03:53:47.548 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-12T03:53:47.549 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-12T03:53:47.549 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-12T03:53:47.549 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-12T03:53:47.550 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : librados2-2:20.2.0-626.g1279775f.el9.x86_64 36/39 2026-02-12T03:53:47.550 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-12T03:53:47.550 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : librbd1-2:20.2.0-626.g1279775f.el9.x86_64 38/39 2026-02-12T03:53:47.608 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T03:53:47.608 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:47.608 INFO:teuthology.orchestra.run.trial103.stdout:Upgraded: 2026-02-12T03:53:47.608 INFO:teuthology.orchestra.run.trial103.stdout: librados2-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.608 INFO:teuthology.orchestra.run.trial103.stdout: librbd1-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.608 INFO:teuthology.orchestra.run.trial103.stdout:Installed: 2026-02-12T03:53:47.608 INFO:teuthology.orchestra.run.trial103.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: ceph-base-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: ceph-common-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: ceph-radosgw-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: ceph-selinux-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: libcephfs-daemon-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: libcephfs-proxy2-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: libcephfs2-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: libradosstriper1-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: librgw2-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: nvme-cli-2.13-1.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: python3-ceph-argparse-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.609 INFO:teuthology.orchestra.run.trial103.stdout: python3-ceph-common-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.610 INFO:teuthology.orchestra.run.trial103.stdout: python3-cephfs-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.610 INFO:teuthology.orchestra.run.trial103.stdout: python3-rados-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.610 INFO:teuthology.orchestra.run.trial103.stdout: python3-rbd-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.610 INFO:teuthology.orchestra.run.trial103.stdout: python3-rgw-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:47.610 INFO:teuthology.orchestra.run.trial103.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-12T03:53:47.610 INFO:teuthology.orchestra.run.trial103.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-12T03:53:47.610 INFO:teuthology.orchestra.run.trial103.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-12T03:53:47.610 INFO:teuthology.orchestra.run.trial103.stdout: re2-1:20211101-20.el9.x86_64 2026-02-12T03:53:47.610 INFO:teuthology.orchestra.run.trial103.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-12T03:53:47.610 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:47.610 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:53:47.653 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install ceph-test 2026-02-12T03:53:47.957 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:16 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:53:48.041 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:53:48.042 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:53:48.043 INFO:teuthology.orchestra.run.trial103.stdout: Package Arch Version Repository Size 2026-02-12T03:53:48.043 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:53:48.043 INFO:teuthology.orchestra.run.trial103.stdout:Installing: 2026-02-12T03:53:48.043 INFO:teuthology.orchestra.run.trial103.stdout: ceph-test x86_64 2:20.2.0-626.g1279775f.el9 ceph 84 M 2026-02-12T03:53:48.043 INFO:teuthology.orchestra.run.trial103.stdout:Installing dependencies: 2026-02-12T03:53:48.043 INFO:teuthology.orchestra.run.trial103.stdout: libcephsqlite x86_64 2:20.2.0-626.g1279775f.el9 ceph 164 k 2026-02-12T03:53:48.043 INFO:teuthology.orchestra.run.trial103.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-12T03:53:48.043 INFO:teuthology.orchestra.run.trial103.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-12T03:53:48.043 INFO:teuthology.orchestra.run.trial103.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-12T03:53:48.043 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:48.044 INFO:teuthology.orchestra.run.trial103.stdout:Transaction Summary 2026-02-12T03:53:48.044 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:53:48.044 INFO:teuthology.orchestra.run.trial103.stdout:Install 5 Packages 2026-02-12T03:53:48.044 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:48.044 INFO:teuthology.orchestra.run.trial103.stdout:Total download size: 85 M 2026-02-12T03:53:48.044 INFO:teuthology.orchestra.run.trial103.stdout:Installed size: 364 M 2026-02-12T03:53:48.044 INFO:teuthology.orchestra.run.trial103.stdout:Downloading Packages: 2026-02-12T03:53:48.440 INFO:teuthology.orchestra.run.trial103.stdout:(1/5): libcephsqlite-20.2.0-626.g1279775f.el9.x 1.8 MB/s | 164 kB 00:00 2026-02-12T03:53:48.502 INFO:teuthology.orchestra.run.trial103.stdout:(2/5): libxslt-1.1.34-12.el9.x86_64.rpm 1.5 MB/s | 233 kB 00:00 2026-02-12T03:53:48.558 INFO:teuthology.orchestra.run.trial103.stdout:(3/5): socat-1.7.4.1-8.el9.x86_64.rpm 2.5 MB/s | 303 kB 00:00 2026-02-12T03:53:48.630 INFO:teuthology.orchestra.run.trial103.stdout:(4/5): xmlstarlet-1.6.1-20.el9.x86_64.rpm 498 kB/s | 64 kB 00:00 2026-02-12T03:53:50.256 INFO:teuthology.orchestra.run.trial103.stdout:(5/5): ceph-test-20.2.0-626.g1279775f.el9.x86_6 44 MB/s | 84 MB 00:01 2026-02-12T03:53:50.258 INFO:teuthology.orchestra.run.trial103.stdout:-------------------------------------------------------------------------------- 2026-02-12T03:53:50.258 INFO:teuthology.orchestra.run.trial103.stdout:Total 38 MB/s | 85 MB 00:02 2026-02-12T03:53:50.287 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction check 2026-02-12T03:53:50.297 INFO:teuthology.orchestra.run.trial103.stdout:Transaction check succeeded. 2026-02-12T03:53:50.298 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction test 2026-02-12T03:53:50.584 INFO:teuthology.orchestra.run.trial103.stdout:Transaction test succeeded. 2026-02-12T03:53:50.585 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction 2026-02-12T03:53:50.899 INFO:teuthology.orchestra.run.trial103.stdout: Preparing : 1/1 2026-02-12T03:53:50.905 INFO:teuthology.orchestra.run.trial103.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 1/5 2026-02-12T03:53:50.910 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 2/5 2026-02-12T03:53:50.918 INFO:teuthology.orchestra.run.trial103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/5 2026-02-12T03:53:50.919 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libcephsqlite-2:20.2.0-626.g1279775f.el9.x86_64 4/5 2026-02-12T03:53:54.713 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: libcephsqlite-2:20.2.0-626.g1279775f.el9.x86_64 4/5 2026-02-12T03:53:54.730 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-test-2:20.2.0-626.g1279775f.el9.x86_64 5/5 2026-02-12T03:53:55.076 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-test-2:20.2.0-626.g1279775f.el9.x86_64 5/5 2026-02-12T03:53:55.076 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-test-2:20.2.0-626.g1279775f.el9.x86_64 1/5 2026-02-12T03:53:55.076 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libcephsqlite-2:20.2.0-626.g1279775f.el9.x86_64 2/5 2026-02-12T03:53:55.076 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 3/5 2026-02-12T03:53:55.077 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 4/5 2026-02-12T03:53:55.128 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/5 2026-02-12T03:53:55.129 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:55.129 INFO:teuthology.orchestra.run.trial103.stdout:Installed: 2026-02-12T03:53:55.129 INFO:teuthology.orchestra.run.trial103.stdout: ceph-test-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:55.129 INFO:teuthology.orchestra.run.trial103.stdout: libcephsqlite-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:53:55.129 INFO:teuthology.orchestra.run.trial103.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-12T03:53:55.129 INFO:teuthology.orchestra.run.trial103.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-12T03:53:55.129 INFO:teuthology.orchestra.run.trial103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-12T03:53:55.129 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:55.129 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:53:55.165 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install ceph 2026-02-12T03:53:55.465 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:24 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:53:55.553 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:53:55.556 INFO:teuthology.orchestra.run.trial103.stdout:====================================================================================== 2026-02-12T03:53:55.556 INFO:teuthology.orchestra.run.trial103.stdout: Package Arch Version Repository Size 2026-02-12T03:53:55.556 INFO:teuthology.orchestra.run.trial103.stdout:====================================================================================== 2026-02-12T03:53:55.556 INFO:teuthology.orchestra.run.trial103.stdout:Installing: 2026-02-12T03:53:55.556 INFO:teuthology.orchestra.run.trial103.stdout: ceph x86_64 2:20.2.0-626.g1279775f.el9 ceph 6.5 k 2026-02-12T03:53:55.556 INFO:teuthology.orchestra.run.trial103.stdout:Installing dependencies: 2026-02-12T03:53:55.556 INFO:teuthology.orchestra.run.trial103.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-12T03:53:55.557 INFO:teuthology.orchestra.run.trial103.stdout: ceph-grafana-dashboards noarch 2:20.2.0-626.g1279775f.el9 ceph-noarch 43 k 2026-02-12T03:53:55.557 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mds x86_64 2:20.2.0-626.g1279775f.el9 ceph 2.3 M 2026-02-12T03:53:55.557 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mgr x86_64 2:20.2.0-626.g1279775f.el9 ceph 962 k 2026-02-12T03:53:55.557 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mgr-modules-core noarch 2:20.2.0-626.g1279775f.el9 ceph-noarch 290 k 2026-02-12T03:53:55.557 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mon x86_64 2:20.2.0-626.g1279775f.el9 ceph 5.0 M 2026-02-12T03:53:55.557 INFO:teuthology.orchestra.run.trial103.stdout: ceph-osd x86_64 2:20.2.0-626.g1279775f.el9 ceph 17 M 2026-02-12T03:53:55.557 INFO:teuthology.orchestra.run.trial103.stdout: ceph-prometheus-alerts noarch 2:20.2.0-626.g1279775f.el9 ceph-noarch 17 k 2026-02-12T03:53:55.557 INFO:teuthology.orchestra.run.trial103.stdout: cephadm noarch 2:20.2.0-626.g1279775f.el9 ceph-noarch 1.0 M 2026-02-12T03:53:55.557 INFO:teuthology.orchestra.run.trial103.stdout: cryptsetup x86_64 2.8.1-2.el9 baseos 350 k 2026-02-12T03:53:55.557 INFO:teuthology.orchestra.run.trial103.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2026-02-12T03:53:55.557 INFO:teuthology.orchestra.run.trial103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2026-02-12T03:53:55.557 INFO:teuthology.orchestra.run.trial103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2026-02-12T03:53:55.558 INFO:teuthology.orchestra.run.trial103.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-12T03:53:55.558 INFO:teuthology.orchestra.run.trial103.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-12T03:53:55.558 INFO:teuthology.orchestra.run.trial103.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-12T03:53:55.558 INFO:teuthology.orchestra.run.trial103.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-12T03:53:55.558 INFO:teuthology.orchestra.run.trial103.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-12T03:53:55.558 INFO:teuthology.orchestra.run.trial103.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-12T03:53:55.558 INFO:teuthology.orchestra.run.trial103.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-12T03:53:55.558 INFO:teuthology.orchestra.run.trial103.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-12T03:53:55.558 INFO:teuthology.orchestra.run.trial103.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-12T03:53:55.558 INFO:teuthology.orchestra.run.trial103.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-12T03:53:55.558 INFO:teuthology.orchestra.run.trial103.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-12T03:53:55.558 INFO:teuthology.orchestra.run.trial103.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-12T03:53:55.558 INFO:teuthology.orchestra.run.trial103.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-12T03:53:55.559 INFO:teuthology.orchestra.run.trial103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-12T03:53:55.559 INFO:teuthology.orchestra.run.trial103.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-12T03:53:55.559 INFO:teuthology.orchestra.run.trial103.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-12T03:53:55.559 INFO:teuthology.orchestra.run.trial103.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-12T03:53:55.559 INFO:teuthology.orchestra.run.trial103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-12T03:53:55.559 INFO:teuthology.orchestra.run.trial103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-12T03:53:55.559 INFO:teuthology.orchestra.run.trial103.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-12T03:53:55.559 INFO:teuthology.orchestra.run.trial103.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-12T03:53:55.559 INFO:teuthology.orchestra.run.trial103.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-12T03:53:55.559 INFO:teuthology.orchestra.run.trial103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-12T03:53:55.559 INFO:teuthology.orchestra.run.trial103.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-12T03:53:55.559 INFO:teuthology.orchestra.run.trial103.stdout: python3-devel x86_64 3.9.25-2.el9 appstream 244 k 2026-02-12T03:53:55.560 INFO:teuthology.orchestra.run.trial103.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-12T03:53:55.560 INFO:teuthology.orchestra.run.trial103.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-12T03:53:55.560 INFO:teuthology.orchestra.run.trial103.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-12T03:53:55.560 INFO:teuthology.orchestra.run.trial103.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-12T03:53:55.560 INFO:teuthology.orchestra.run.trial103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-12T03:53:55.560 INFO:teuthology.orchestra.run.trial103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-12T03:53:55.560 INFO:teuthology.orchestra.run.trial103.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-12T03:53:55.560 INFO:teuthology.orchestra.run.trial103.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-12T03:53:55.560 INFO:teuthology.orchestra.run.trial103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-12T03:53:55.560 INFO:teuthology.orchestra.run.trial103.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-12T03:53:55.560 INFO:teuthology.orchestra.run.trial103.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-12T03:53:55.560 INFO:teuthology.orchestra.run.trial103.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-12T03:53:55.560 INFO:teuthology.orchestra.run.trial103.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-12T03:53:55.561 INFO:teuthology.orchestra.run.trial103.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-12T03:53:55.561 INFO:teuthology.orchestra.run.trial103.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-12T03:53:55.561 INFO:teuthology.orchestra.run.trial103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-12T03:53:55.561 INFO:teuthology.orchestra.run.trial103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-12T03:53:55.561 INFO:teuthology.orchestra.run.trial103.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-12T03:53:55.561 INFO:teuthology.orchestra.run.trial103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-12T03:53:55.561 INFO:teuthology.orchestra.run.trial103.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-12T03:53:55.561 INFO:teuthology.orchestra.run.trial103.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-12T03:53:55.561 INFO:teuthology.orchestra.run.trial103.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-12T03:53:55.561 INFO:teuthology.orchestra.run.trial103.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-12T03:53:55.561 INFO:teuthology.orchestra.run.trial103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-12T03:53:55.561 INFO:teuthology.orchestra.run.trial103.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-12T03:53:55.561 INFO:teuthology.orchestra.run.trial103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-12T03:53:55.562 INFO:teuthology.orchestra.run.trial103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-12T03:53:55.562 INFO:teuthology.orchestra.run.trial103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-12T03:53:55.562 INFO:teuthology.orchestra.run.trial103.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-12T03:53:55.562 INFO:teuthology.orchestra.run.trial103.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-12T03:53:55.562 INFO:teuthology.orchestra.run.trial103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-12T03:53:55.562 INFO:teuthology.orchestra.run.trial103.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-12T03:53:55.562 INFO:teuthology.orchestra.run.trial103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-12T03:53:55.562 INFO:teuthology.orchestra.run.trial103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-12T03:53:55.562 INFO:teuthology.orchestra.run.trial103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-12T03:53:55.562 INFO:teuthology.orchestra.run.trial103.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-12T03:53:55.562 INFO:teuthology.orchestra.run.trial103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-12T03:53:55.562 INFO:teuthology.orchestra.run.trial103.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-12T03:53:55.562 INFO:teuthology.orchestra.run.trial103.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-12T03:53:55.563 INFO:teuthology.orchestra.run.trial103.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-12T03:53:55.563 INFO:teuthology.orchestra.run.trial103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-12T03:53:55.563 INFO:teuthology.orchestra.run.trial103.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-12T03:53:55.563 INFO:teuthology.orchestra.run.trial103.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-12T03:53:55.563 INFO:teuthology.orchestra.run.trial103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-12T03:53:55.563 INFO:teuthology.orchestra.run.trial103.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-12T03:53:55.563 INFO:teuthology.orchestra.run.trial103.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-12T03:53:55.563 INFO:teuthology.orchestra.run.trial103.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-12T03:53:55.563 INFO:teuthology.orchestra.run.trial103.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-12T03:53:55.563 INFO:teuthology.orchestra.run.trial103.stdout:Installing weak dependencies: 2026-02-12T03:53:55.563 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mgr-cephadm noarch 2:20.2.0-626.g1279775f.el9 ceph-noarch 173 k 2026-02-12T03:53:55.563 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mgr-dashboard noarch 2:20.2.0-626.g1279775f.el9 ceph-noarch 11 M 2026-02-12T03:53:55.563 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mgr-diskprediction-local noarch 2:20.2.0-626.g1279775f.el9 ceph-noarch 7.4 M 2026-02-12T03:53:55.564 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mgr-k8sevents noarch 2:20.2.0-626.g1279775f.el9 ceph-noarch 22 k 2026-02-12T03:53:55.564 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mgr-rook noarch 2:20.2.0-626.g1279775f.el9 ceph-noarch 50 k 2026-02-12T03:53:55.564 INFO:teuthology.orchestra.run.trial103.stdout: ceph-volume noarch 2:20.2.0-626.g1279775f.el9 ceph-noarch 298 k 2026-02-12T03:53:55.564 INFO:teuthology.orchestra.run.trial103.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-12T03:53:55.564 INFO:teuthology.orchestra.run.trial103.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-12T03:53:55.564 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:55.564 INFO:teuthology.orchestra.run.trial103.stdout:Transaction Summary 2026-02-12T03:53:55.564 INFO:teuthology.orchestra.run.trial103.stdout:====================================================================================== 2026-02-12T03:53:55.564 INFO:teuthology.orchestra.run.trial103.stdout:Install 95 Packages 2026-02-12T03:53:55.564 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:53:55.565 INFO:teuthology.orchestra.run.trial103.stdout:Total download size: 101 M 2026-02-12T03:53:55.565 INFO:teuthology.orchestra.run.trial103.stdout:Installed size: 561 M 2026-02-12T03:53:55.565 INFO:teuthology.orchestra.run.trial103.stdout:Downloading Packages: 2026-02-12T03:53:55.992 INFO:teuthology.orchestra.run.trial103.stdout:(1/95): ceph-20.2.0-626.g1279775f.el9.x86_64.rp 37 kB/s | 6.5 kB 00:00 2026-02-12T03:53:57.156 INFO:teuthology.orchestra.run.trial103.stdout:(2/95): ceph-mgr-20.2.0-626.g1279775f.el9.x86_6 718 kB/s | 962 kB 00:01 2026-02-12T03:53:57.653 INFO:teuthology.orchestra.run.trial103.stdout:(3/95): ceph-mon-20.2.0-626.g1279775f.el9.x86_6 3.0 MB/s | 5.0 MB 00:01 2026-02-12T03:53:57.674 INFO:teuthology.orchestra.run.trial103.stdout:(4/95): ceph-grafana-dashboards-20.2.0-626.g127 2.0 MB/s | 43 kB 00:00 2026-02-12T03:53:57.698 INFO:teuthology.orchestra.run.trial103.stdout:(5/95): ceph-mgr-cephadm-20.2.0-626.g1279775f.e 7.2 MB/s | 173 kB 00:00 2026-02-12T03:53:57.754 INFO:teuthology.orchestra.run.trial103.stdout:(6/95): ceph-mds-20.2.0-626.g1279775f.el9.x86_6 1.2 MB/s | 2.3 MB 00:01 2026-02-12T03:53:58.106 INFO:teuthology.orchestra.run.trial103.stdout:(7/95): ceph-mgr-dashboard-20.2.0-626.g1279775f 26 MB/s | 11 MB 00:00 2026-02-12T03:53:58.123 INFO:teuthology.orchestra.run.trial103.stdout:(8/95): ceph-mgr-k8sevents-20.2.0-626.g1279775f 1.3 MB/s | 22 kB 00:00 2026-02-12T03:53:58.145 INFO:teuthology.orchestra.run.trial103.stdout:(9/95): ceph-mgr-modules-core-20.2.0-626.g12797 13 MB/s | 290 kB 00:00 2026-02-12T03:53:58.163 INFO:teuthology.orchestra.run.trial103.stdout:(10/95): ceph-mgr-rook-20.2.0-626.g1279775f.el9 2.7 MB/s | 50 kB 00:00 2026-02-12T03:53:58.176 INFO:teuthology.orchestra.run.trial103.stdout:(11/95): ceph-prometheus-alerts-20.2.0-626.g127 1.2 MB/s | 17 kB 00:00 2026-02-12T03:53:58.196 INFO:teuthology.orchestra.run.trial103.stdout:(12/95): ceph-volume-20.2.0-626.g1279775f.el9.n 15 MB/s | 298 kB 00:00 2026-02-12T03:53:58.249 INFO:teuthology.orchestra.run.trial103.stdout:(13/95): cephadm-20.2.0-626.g1279775f.el9.noarc 19 MB/s | 1.0 MB 00:00 2026-02-12T03:53:58.418 INFO:teuthology.orchestra.run.trial103.stdout:(14/95): cryptsetup-2.8.1-2.el9.x86_64.rpm 2.0 MB/s | 350 kB 00:00 2026-02-12T03:53:58.443 INFO:teuthology.orchestra.run.trial103.stdout:(15/95): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.5 MB/s | 40 kB 00:00 2026-02-12T03:53:58.470 INFO:teuthology.orchestra.run.trial103.stdout:(16/95): libconfig-1.7.2-9.el9.x86_64.rpm 2.6 MB/s | 72 kB 00:00 2026-02-12T03:53:58.533 INFO:teuthology.orchestra.run.trial103.stdout:(17/95): libgfortran-11.5.0-14.el9.x86_64.rpm 12 MB/s | 794 kB 00:00 2026-02-12T03:53:58.561 INFO:teuthology.orchestra.run.trial103.stdout:(18/95): libquadmath-11.5.0-14.el9.x86_64.rpm 6.4 MB/s | 184 kB 00:00 2026-02-12T03:53:58.590 INFO:teuthology.orchestra.run.trial103.stdout:(19/95): python3-cffi-1.14.5-5.el9.x86_64.rpm 8.7 MB/s | 253 kB 00:00 2026-02-12T03:53:58.640 INFO:teuthology.orchestra.run.trial103.stdout:(20/95): python3-cryptography-36.0.1-5.el9.x86_ 25 MB/s | 1.2 MB 00:00 2026-02-12T03:53:58.667 INFO:teuthology.orchestra.run.trial103.stdout:(21/95): python3-ply-3.11-14.el9.noarch.rpm 3.9 MB/s | 106 kB 00:00 2026-02-12T03:53:58.695 INFO:teuthology.orchestra.run.trial103.stdout:(22/95): python3-pycparser-2.20-6.el9.noarch.rp 4.8 MB/s | 135 kB 00:00 2026-02-12T03:53:58.722 INFO:teuthology.orchestra.run.trial103.stdout:(23/95): python3-pyparsing-2.4.7-9.el9.noarch.r 5.4 MB/s | 150 kB 00:00 2026-02-12T03:53:58.749 INFO:teuthology.orchestra.run.trial103.stdout:(24/95): python3-requests-2.25.1-10.el9.noarch. 4.5 MB/s | 126 kB 00:00 2026-02-12T03:53:58.777 INFO:teuthology.orchestra.run.trial103.stdout:(25/95): python3-urllib3-1.26.5-7.el9.noarch.rp 7.7 MB/s | 218 kB 00:00 2026-02-12T03:53:58.805 INFO:teuthology.orchestra.run.trial103.stdout:(26/95): unzip-6.0-59.el9.x86_64.rpm 6.4 MB/s | 182 kB 00:00 2026-02-12T03:53:58.832 INFO:teuthology.orchestra.run.trial103.stdout:(27/95): zip-3.0-35.el9.x86_64.rpm 9.5 MB/s | 266 kB 00:00 2026-02-12T03:53:58.903 INFO:teuthology.orchestra.run.trial103.stdout:(28/95): flexiblas-3.0.4-8.el9.x86_64.rpm 466 kB/s | 33 kB 00:00 2026-02-12T03:53:59.032 INFO:teuthology.orchestra.run.trial103.stdout:(29/95): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 23 MB/s | 3.0 MB 00:00 2026-02-12T03:53:59.054 INFO:teuthology.orchestra.run.trial103.stdout:(30/95): flexiblas-openblas-openmp-3.0.4-8.el9. 838 kB/s | 18 kB 00:00 2026-02-12T03:53:59.077 INFO:teuthology.orchestra.run.trial103.stdout:(31/95): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 10 MB/s | 246 kB 00:00 2026-02-12T03:53:59.101 INFO:teuthology.orchestra.run.trial103.stdout:(32/95): lua-5.4.4-4.el9.x86_64.rpm 7.9 MB/s | 188 kB 00:00 2026-02-12T03:53:59.123 INFO:teuthology.orchestra.run.trial103.stdout:(33/95): openblas-0.3.29-1.el9.x86_64.rpm 1.9 MB/s | 42 kB 00:00 2026-02-12T03:53:59.174 INFO:teuthology.orchestra.run.trial103.stdout:(34/95): openblas-openmp-0.3.29-1.el9.x86_64.rp 103 MB/s | 5.3 MB 00:00 2026-02-12T03:53:59.200 INFO:teuthology.orchestra.run.trial103.stdout:(35/95): protobuf-3.14.0-17.el9.x86_64.rpm 39 MB/s | 1.0 MB 00:00 2026-02-12T03:53:59.256 INFO:teuthology.orchestra.run.trial103.stdout:(36/95): python3-babel-2.9.1-2.el9.noarch.rpm 107 MB/s | 6.0 MB 00:00 2026-02-12T03:53:59.280 INFO:teuthology.orchestra.run.trial103.stdout:(37/95): python3-devel-3.9.25-2.el9.x86_64.rpm 9.8 MB/s | 244 kB 00:00 2026-02-12T03:53:59.305 INFO:teuthology.orchestra.run.trial103.stdout:(38/95): python3-jinja2-2.11.3-8.el9.noarch.rpm 10 MB/s | 249 kB 00:00 2026-02-12T03:53:59.327 INFO:teuthology.orchestra.run.trial103.stdout:(39/95): python3-jmespath-1.0.1-1.el9.noarch.rp 2.1 MB/s | 48 kB 00:00 2026-02-12T03:53:59.350 INFO:teuthology.orchestra.run.trial103.stdout:(40/95): python3-libstoragemgmt-1.10.1-1.el9.x8 7.5 MB/s | 177 kB 00:00 2026-02-12T03:53:59.377 INFO:teuthology.orchestra.run.trial103.stdout:(41/95): python3-lxml-4.6.5-3.el9.x86_64.rpm 45 MB/s | 1.2 MB 00:00 2026-02-12T03:53:59.398 INFO:teuthology.orchestra.run.trial103.stdout:(42/95): ceph-mgr-diskprediction-local-20.2.0-6 4.5 MB/s | 7.4 MB 00:01 2026-02-12T03:53:59.399 INFO:teuthology.orchestra.run.trial103.stdout:(43/95): python3-markupsafe-1.1.1-12.el9.x86_64 1.5 MB/s | 35 kB 00:00 2026-02-12T03:53:59.423 INFO:teuthology.orchestra.run.trial103.stdout:(44/95): python3-numpy-f2py-1.23.5-2.el9.x86_64 18 MB/s | 442 kB 00:00 2026-02-12T03:53:59.446 INFO:teuthology.orchestra.run.trial103.stdout:(45/95): python3-packaging-20.9-5.el9.noarch.rp 3.3 MB/s | 77 kB 00:00 2026-02-12T03:53:59.469 INFO:teuthology.orchestra.run.trial103.stdout:(46/95): python3-protobuf-3.14.0-17.el9.noarch. 11 MB/s | 267 kB 00:00 2026-02-12T03:53:59.493 INFO:teuthology.orchestra.run.trial103.stdout:(47/95): python3-pyasn1-0.4.8-6.el9.noarch.rpm 6.5 MB/s | 159 kB 00:00 2026-02-12T03:53:59.518 INFO:teuthology.orchestra.run.trial103.stdout:(48/95): python3-pyasn1-modules-0.4.8-6.el9.noa 11 MB/s | 279 kB 00:00 2026-02-12T03:53:59.540 INFO:teuthology.orchestra.run.trial103.stdout:(49/95): python3-requests-oauthlib-1.3.0-12.el9 2.3 MB/s | 54 kB 00:00 2026-02-12T03:53:59.594 INFO:teuthology.orchestra.run.trial103.stdout:(50/95): python3-numpy-1.23.5-2.el9.x86_64.rpm 31 MB/s | 6.1 MB 00:00 2026-02-12T03:53:59.617 INFO:teuthology.orchestra.run.trial103.stdout:(51/95): python3-toml-0.10.2-6.el9.noarch.rpm 1.8 MB/s | 42 kB 00:00 2026-02-12T03:53:59.640 INFO:teuthology.orchestra.run.trial103.stdout:(52/95): xmlsec1-1.2.29-13.el9.x86_64.rpm 8.1 MB/s | 189 kB 00:00 2026-02-12T03:53:59.663 INFO:teuthology.orchestra.run.trial103.stdout:(53/95): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 3.9 MB/s | 90 kB 00:00 2026-02-12T03:53:59.693 INFO:teuthology.orchestra.run.trial103.stdout:(54/95): python3-scipy-1.9.3-2.el9.x86_64.rpm 126 MB/s | 19 MB 00:00 2026-02-12T03:53:59.710 INFO:teuthology.orchestra.run.trial103.stdout:(55/95): lua-devel-5.4.4-4.el9.x86_64.rpm 476 kB/s | 22 kB 00:00 2026-02-12T03:53:59.751 INFO:teuthology.orchestra.run.trial103.stdout:(56/95): protobuf-compiler-3.14.0-17.el9.x86_64 14 MB/s | 862 kB 00:00 2026-02-12T03:53:59.779 INFO:teuthology.orchestra.run.trial103.stdout:(57/95): abseil-cpp-20211102.0-4.el9.x86_64.rpm 7.8 MB/s | 551 kB 00:00 2026-02-12T03:53:59.780 INFO:teuthology.orchestra.run.trial103.stdout:(58/95): grpc-data-1.46.7-10.el9.noarch.rpm 669 kB/s | 19 kB 00:00 2026-02-12T03:53:59.790 INFO:teuthology.orchestra.run.trial103.stdout:(59/95): luarocks-3.9.2-5.el9.noarch.rpm 13 MB/s | 151 kB 00:00 2026-02-12T03:53:59.800 INFO:teuthology.orchestra.run.trial103.stdout:(60/95): python3-autocommand-2.2.2-8.el9.noarch 2.9 MB/s | 29 kB 00:00 2026-02-12T03:53:59.810 INFO:teuthology.orchestra.run.trial103.stdout:(61/95): python3-backports-tarfile-1.2.0-1.el9. 5.8 MB/s | 60 kB 00:00 2026-02-12T03:53:59.820 INFO:teuthology.orchestra.run.trial103.stdout:(62/95): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 4.2 MB/s | 43 kB 00:00 2026-02-12T03:53:59.830 INFO:teuthology.orchestra.run.trial103.stdout:(63/95): python3-cachetools-4.2.4-1.el9.noarch. 3.3 MB/s | 32 kB 00:00 2026-02-12T03:53:59.832 INFO:teuthology.orchestra.run.trial103.stdout:(64/95): python3-asyncssh-2.13.2-5.el9.noarch.r 10 MB/s | 548 kB 00:00 2026-02-12T03:53:59.840 INFO:teuthology.orchestra.run.trial103.stdout:(65/95): python3-certifi-2023.05.07-4.el9.noarc 1.4 MB/s | 14 kB 00:00 2026-02-12T03:53:59.845 INFO:teuthology.orchestra.run.trial103.stdout:(66/95): python3-cheroot-10.0.1-4.el9.noarch.rp 14 MB/s | 173 kB 00:00 2026-02-12T03:53:59.856 INFO:teuthology.orchestra.run.trial103.stdout:(67/95): python3-cherrypy-18.6.1-2.el9.noarch.r 22 MB/s | 358 kB 00:00 2026-02-12T03:53:59.858 INFO:teuthology.orchestra.run.trial103.stdout:(68/95): python3-google-auth-2.45.0-1.el9.noarc 19 MB/s | 254 kB 00:00 2026-02-12T03:53:59.871 INFO:teuthology.orchestra.run.trial103.stdout:(69/95): python3-grpcio-tools-1.46.7-10.el9.x86 11 MB/s | 144 kB 00:00 2026-02-12T03:53:59.884 INFO:teuthology.orchestra.run.trial103.stdout:(70/95): python3-influxdb-5.3.1-1.el9.noarch.rp 11 MB/s | 139 kB 00:00 2026-02-12T03:53:59.887 INFO:teuthology.orchestra.run.trial103.stdout:(71/95): python3-grpcio-1.46.7-10.el9.x86_64.rp 66 MB/s | 2.0 MB 00:00 2026-02-12T03:53:59.895 INFO:teuthology.orchestra.run.trial103.stdout:(72/95): python3-isodate-0.6.1-3.el9.noarch.rpm 4.8 MB/s | 56 kB 00:00 2026-02-12T03:53:59.896 INFO:teuthology.orchestra.run.trial103.stdout:(73/95): python3-jaraco-8.2.1-3.el9.noarch.rpm 1.1 MB/s | 11 kB 00:00 2026-02-12T03:53:59.906 INFO:teuthology.orchestra.run.trial103.stdout:(74/95): python3-jaraco-classes-3.2.1-5.el9.noa 1.6 MB/s | 18 kB 00:00 2026-02-12T03:53:59.907 INFO:teuthology.orchestra.run.trial103.stdout:(75/95): python3-jaraco-collections-3.0.0-8.el9 2.3 MB/s | 23 kB 00:00 2026-02-12T03:53:59.916 INFO:teuthology.orchestra.run.trial103.stdout:(76/95): python3-jaraco-functools-3.5.0-2.el9.n 2.0 MB/s | 19 kB 00:00 2026-02-12T03:53:59.917 INFO:teuthology.orchestra.run.trial103.stdout:(77/95): python3-jaraco-context-6.0.1-3.el9.noa 1.8 MB/s | 20 kB 00:00 2026-02-12T03:53:59.926 INFO:teuthology.orchestra.run.trial103.stdout:(78/95): python3-jaraco-text-4.0.0-2.el9.noarch 2.7 MB/s | 26 kB 00:00 2026-02-12T03:53:59.936 INFO:teuthology.orchestra.run.trial103.stdout:(79/95): python3-more-itertools-8.12.0-2.el9.no 7.5 MB/s | 79 kB 00:00 2026-02-12T03:53:59.939 INFO:teuthology.orchestra.run.trial103.stdout:(80/95): python3-kubernetes-26.1.0-3.el9.noarch 45 MB/s | 1.0 MB 00:00 2026-02-12T03:53:59.947 INFO:teuthology.orchestra.run.trial103.stdout:(81/95): python3-msgpack-1.0.3-2.el9.x86_64.rpm 7.9 MB/s | 86 kB 00:00 2026-02-12T03:53:59.950 INFO:teuthology.orchestra.run.trial103.stdout:(82/95): python3-natsort-7.1.1-5.el9.noarch.rpm 5.1 MB/s | 58 kB 00:00 2026-02-12T03:53:59.957 INFO:teuthology.orchestra.run.trial103.stdout:(83/95): python3-portend-3.1.0-2.el9.noarch.rpm 1.7 MB/s | 16 kB 00:00 2026-02-12T03:53:59.962 INFO:teuthology.orchestra.run.trial103.stdout:(84/95): python3-pyOpenSSL-21.0.0-1.el9.noarch. 7.7 MB/s | 90 kB 00:00 2026-02-12T03:53:59.967 INFO:teuthology.orchestra.run.trial103.stdout:(85/95): python3-repoze-lru-0.7-16.el9.noarch.r 3.1 MB/s | 31 kB 00:00 2026-02-12T03:53:59.974 INFO:teuthology.orchestra.run.trial103.stdout:(86/95): python3-routes-2.5.1-5.el9.noarch.rpm 15 MB/s | 188 kB 00:00 2026-02-12T03:53:59.976 INFO:teuthology.orchestra.run.trial103.stdout:(87/95): python3-rsa-4.9-2.el9.noarch.rpm 6.0 MB/s | 59 kB 00:00 2026-02-12T03:53:59.986 INFO:teuthology.orchestra.run.trial103.stdout:(88/95): python3-tempora-5.0.0-2.el9.noarch.rpm 3.5 MB/s | 36 kB 00:00 2026-02-12T03:53:59.987 INFO:teuthology.orchestra.run.trial103.stdout:(89/95): python3-saml-1.16.0-1.el9.noarch.rpm 9.8 MB/s | 125 kB 00:00 2026-02-12T03:53:59.997 INFO:teuthology.orchestra.run.trial103.stdout:(90/95): python3-typing-extensions-4.15.0-1.el9 8.0 MB/s | 86 kB 00:00 2026-02-12T03:53:59.998 INFO:teuthology.orchestra.run.trial103.stdout:(91/95): python3-websocket-client-1.2.3-2.el9.n 8.0 MB/s | 90 kB 00:00 2026-02-12T03:54:00.007 INFO:teuthology.orchestra.run.trial103.stdout:(92/95): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 4.8 MB/s | 48 kB 00:00 2026-02-12T03:54:00.009 INFO:teuthology.orchestra.run.trial103.stdout:(93/95): python3-xmltodict-0.12.0-15.el9.noarch 2.0 MB/s | 22 kB 00:00 2026-02-12T03:54:00.017 INFO:teuthology.orchestra.run.trial103.stdout:(94/95): python3-zc-lockfile-2.0-10.el9.noarch. 1.9 MB/s | 20 kB 00:00 2026-02-12T03:54:00.040 INFO:teuthology.orchestra.run.trial103.stdout:(95/95): ceph-osd-20.2.0-626.g1279775f.el9.x86_ 5.9 MB/s | 17 MB 00:02 2026-02-12T03:54:00.041 INFO:teuthology.orchestra.run.trial103.stdout:-------------------------------------------------------------------------------- 2026-02-12T03:54:00.041 INFO:teuthology.orchestra.run.trial103.stdout:Total 23 MB/s | 101 MB 00:04 2026-02-12T03:54:00.473 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction check 2026-02-12T03:54:00.512 INFO:teuthology.orchestra.run.trial103.stdout:Transaction check succeeded. 2026-02-12T03:54:00.512 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction test 2026-02-12T03:54:00.944 INFO:teuthology.orchestra.run.trial103.stdout:Transaction test succeeded. 2026-02-12T03:54:00.944 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction 2026-02-12T03:54:01.411 INFO:teuthology.orchestra.run.trial103.stdout: Preparing : 1/1 2026-02-12T03:54:01.412 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/95 2026-02-12T03:54:01.416 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/95 2026-02-12T03:54:01.438 INFO:teuthology.orchestra.run.trial103.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/95 2026-02-12T03:54:01.471 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/95 2026-02-12T03:54:01.481 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/95 2026-02-12T03:54:01.486 INFO:teuthology.orchestra.run.trial103.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 6/95 2026-02-12T03:54:01.489 INFO:teuthology.orchestra.run.trial103.stdout: Installing : lua-5.4.4-4.el9.x86_64 7/95 2026-02-12T03:54:01.493 INFO:teuthology.orchestra.run.trial103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 8/95 2026-02-12T03:54:01.517 INFO:teuthology.orchestra.run.trial103.stdout: Installing : unzip-6.0-59.el9.x86_64 9/95 2026-02-12T03:54:01.529 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 10/95 2026-02-12T03:54:01.532 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 11/95 2026-02-12T03:54:01.538 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 12/95 2026-02-12T03:54:01.540 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 13/95 2026-02-12T03:54:01.551 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 14/95 2026-02-12T03:54:01.558 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 15/95 2026-02-12T03:54:01.582 INFO:teuthology.orchestra.run.trial103.stdout: Installing : zip-3.0-35.el9.x86_64 16/95 2026-02-12T03:54:01.585 INFO:teuthology.orchestra.run.trial103.stdout: Installing : luarocks-3.9.2-5.el9.noarch 17/95 2026-02-12T03:54:01.592 INFO:teuthology.orchestra.run.trial103.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 18/95 2026-02-12T03:54:01.643 INFO:teuthology.orchestra.run.trial103.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 19/95 2026-02-12T03:54:01.655 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 20/95 2026-02-12T03:54:01.658 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 21/95 2026-02-12T03:54:01.662 INFO:teuthology.orchestra.run.trial103.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 22/95 2026-02-12T03:54:01.668 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 23/95 2026-02-12T03:54:01.672 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 24/95 2026-02-12T03:54:01.676 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 25/95 2026-02-12T03:54:01.690 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 26/95 2026-02-12T03:54:01.695 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 27/95 2026-02-12T03:54:01.700 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 28/95 2026-02-12T03:54:01.711 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 29/95 2026-02-12T03:54:01.720 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 30/95 2026-02-12T03:54:01.726 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 31/95 2026-02-12T03:54:01.746 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 32/95 2026-02-12T03:54:01.757 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 33/95 2026-02-12T03:54:01.772 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 34/95 2026-02-12T03:54:01.777 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 35/95 2026-02-12T03:54:01.785 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 36/95 2026-02-12T03:54:01.825 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 37/95 2026-02-12T03:54:02.149 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 38/95 2026-02-12T03:54:02.159 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 39/95 2026-02-12T03:54:02.163 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 40/95 2026-02-12T03:54:02.169 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 41/95 2026-02-12T03:54:02.171 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 42/95 2026-02-12T03:54:02.199 INFO:teuthology.orchestra.run.trial103.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 43/95 2026-02-12T03:54:02.243 INFO:teuthology.orchestra.run.trial103.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 44/95 2026-02-12T03:54:02.249 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 45/95 2026-02-12T03:54:02.253 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 46/95 2026-02-12T03:54:02.259 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 47/95 2026-02-12T03:54:02.262 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 48/95 2026-02-12T03:54:02.270 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 49/95 2026-02-12T03:54:02.273 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 50/95 2026-02-12T03:54:02.301 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 51/95 2026-02-12T03:54:02.311 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 52/95 2026-02-12T03:54:02.317 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 53/95 2026-02-12T03:54:02.324 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 54/95 2026-02-12T03:54:02.359 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 55/95 2026-02-12T03:54:02.584 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-devel-3.9.25-2.el9.x86_64 56/95 2026-02-12T03:54:02.607 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 57/95 2026-02-12T03:54:02.611 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 58/95 2026-02-12T03:54:02.671 INFO:teuthology.orchestra.run.trial103.stdout: Installing : openblas-0.3.29-1.el9.x86_64 59/95 2026-02-12T03:54:02.673 INFO:teuthology.orchestra.run.trial103.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 60/95 2026-02-12T03:54:02.694 INFO:teuthology.orchestra.run.trial103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 61/95 2026-02-12T03:54:03.020 INFO:teuthology.orchestra.run.trial103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 62/95 2026-02-12T03:54:03.093 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 63/95 2026-02-12T03:54:03.765 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 64/95 2026-02-12T03:54:03.782 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 65/95 2026-02-12T03:54:03.792 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 66/95 2026-02-12T03:54:03.799 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 67/95 2026-02-12T03:54:03.812 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-ply-3.11-14.el9.noarch 68/95 2026-02-12T03:54:03.828 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 69/95 2026-02-12T03:54:03.903 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 70/95 2026-02-12T03:54:03.912 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 71/95 2026-02-12T03:54:03.935 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 72/95 2026-02-12T03:54:03.965 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 73/95 2026-02-12T03:54:04.016 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 74/95 2026-02-12T03:54:04.021 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 75/95 2026-02-12T03:54:04.025 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 76/95 2026-02-12T03:54:04.026 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 77/95 2026-02-12T03:54:04.044 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 78/95 2026-02-12T03:54:04.044 INFO:teuthology.orchestra.run.trial103.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-12T03:54:04.044 INFO:teuthology.orchestra.run.trial103.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-12T03:54:04.044 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:04.051 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 78/95 2026-02-12T03:54:04.076 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 78/95 2026-02-12T03:54:04.076 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-12T03:54:04.076 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:04.112 INFO:teuthology.orchestra.run.trial103.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 79/95 2026-02-12T03:54:04.175 INFO:teuthology.orchestra.run.trial103.stdout: Installing : cryptsetup-2.8.1-2.el9.x86_64 80/95 2026-02-12T03:54:04.177 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-volume-2:20.2.0-626.g1279775f.el9.noarch 81/95 2026-02-12T03:54:04.190 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-volume-2:20.2.0-626.g1279775f.el9.noarch 81/95 2026-02-12T03:54:04.191 INFO:teuthology.orchestra.run.trial103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T03:54:04.191 INFO:teuthology.orchestra.run.trial103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-12T03:54:04.191 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:04.865 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-osd-2:20.2.0-626.g1279775f.el9.x86_64 82/95 2026-02-12T03:54:04.889 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-osd-2:20.2.0-626.g1279775f.el9.x86_64 82/95 2026-02-12T03:54:04.889 INFO:teuthology.orchestra.run.trial103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T03:54:04.889 INFO:teuthology.orchestra.run.trial103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-12T03:54:04.889 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-12T03:54:04.889 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-12T03:54:04.890 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:04.948 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: cephadm-2:20.2.0-626.g1279775f.el9.noarch 83/95 2026-02-12T03:54:04.950 INFO:teuthology.orchestra.run.trial103.stdout: Installing : cephadm-2:20.2.0-626.g1279775f.el9.noarch 83/95 2026-02-12T03:54:04.957 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-prometheus-alerts-2:20.2.0-626.g1279775f.el9. 84/95 2026-02-12T03:54:04.980 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-grafana-dashboards-2:20.2.0-626.g1279775f.el9 85/95 2026-02-12T03:54:04.982 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-mgr-cephadm-2:20.2.0-626.g1279775f.el9.noarch 86/95 2026-02-12T03:54:06.129 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.2.0-626.g1279775f.el9.noarch 86/95 2026-02-12T03:54:06.134 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-mgr-dashboard-2:20.2.0-626.g1279775f.el9.noar 87/95 2026-02-12T03:54:06.586 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.2.0-626.g1279775f.el9.noar 87/95 2026-02-12T03:54:06.587 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-mgr-diskprediction-local-2:20.2.0-626.g127977 88/95 2026-02-12T03:54:06.600 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.2.0-626.g127977 88/95 2026-02-12T03:54:06.601 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-mgr-k8sevents-2:20.2.0-626.g1279775f.el9.noar 89/95 2026-02-12T03:54:06.658 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.2.0-626.g1279775f.el9.noar 89/95 2026-02-12T03:54:06.700 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-mgr-modules-core-2:20.2.0-626.g1279775f.el9.n 90/95 2026-02-12T03:54:06.701 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-mgr-2:20.2.0-626.g1279775f.el9.x86_64 91/95 2026-02-12T03:54:06.723 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-mgr-2:20.2.0-626.g1279775f.el9.x86_64 91/95 2026-02-12T03:54:06.723 INFO:teuthology.orchestra.run.trial103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T03:54:06.723 INFO:teuthology.orchestra.run.trial103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-12T03:54:06.723 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-12T03:54:06.723 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-12T03:54:06.723 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:06.736 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-mgr-rook-2:20.2.0-626.g1279775f.el9.noarch 92/95 2026-02-12T03:54:06.959 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-mgr-rook-2:20.2.0-626.g1279775f.el9.noarch 92/95 2026-02-12T03:54:06.960 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-mon-2:20.2.0-626.g1279775f.el9.x86_64 93/95 2026-02-12T03:54:06.982 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-mon-2:20.2.0-626.g1279775f.el9.x86_64 93/95 2026-02-12T03:54:06.982 INFO:teuthology.orchestra.run.trial103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T03:54:06.982 INFO:teuthology.orchestra.run.trial103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-12T03:54:06.982 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-12T03:54:06.982 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-12T03:54:06.982 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:07.081 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-mds-2:20.2.0-626.g1279775f.el9.x86_64 94/95 2026-02-12T03:54:07.101 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-mds-2:20.2.0-626.g1279775f.el9.x86_64 94/95 2026-02-12T03:54:07.102 INFO:teuthology.orchestra.run.trial103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T03:54:07.102 INFO:teuthology.orchestra.run.trial103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-12T03:54:07.102 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-12T03:54:07.102 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-12T03:54:07.102 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:07.120 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-2:20.2.0-626.g1279775f.el9.x86_64 95/95 2026-02-12T03:54:07.822 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-2:20.2.0-626.g1279775f.el9.x86_64 95/95 2026-02-12T03:54:07.823 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-2:20.2.0-626.g1279775f.el9.x86_64 1/95 2026-02-12T03:54:07.823 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-mds-2:20.2.0-626.g1279775f.el9.x86_64 2/95 2026-02-12T03:54:07.823 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-mgr-2:20.2.0-626.g1279775f.el9.x86_64 3/95 2026-02-12T03:54:07.823 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-mon-2:20.2.0-626.g1279775f.el9.x86_64 4/95 2026-02-12T03:54:07.823 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-osd-2:20.2.0-626.g1279775f.el9.x86_64 5/95 2026-02-12T03:54:07.823 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-grafana-dashboards-2:20.2.0-626.g1279775f.el9 6/95 2026-02-12T03:54:07.823 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-mgr-cephadm-2:20.2.0-626.g1279775f.el9.noarch 7/95 2026-02-12T03:54:07.823 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-mgr-dashboard-2:20.2.0-626.g1279775f.el9.noar 8/95 2026-02-12T03:54:07.823 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.2.0-626.g127977 9/95 2026-02-12T03:54:07.823 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-mgr-k8sevents-2:20.2.0-626.g1279775f.el9.noar 10/95 2026-02-12T03:54:07.823 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-mgr-modules-core-2:20.2.0-626.g1279775f.el9.n 11/95 2026-02-12T03:54:07.823 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-mgr-rook-2:20.2.0-626.g1279775f.el9.noarch 12/95 2026-02-12T03:54:07.824 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-prometheus-alerts-2:20.2.0-626.g1279775f.el9. 13/95 2026-02-12T03:54:07.824 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-volume-2:20.2.0-626.g1279775f.el9.noarch 14/95 2026-02-12T03:54:07.824 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : cephadm-2:20.2.0-626.g1279775f.el9.noarch 15/95 2026-02-12T03:54:07.824 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : cryptsetup-2.8.1-2.el9.x86_64 16/95 2026-02-12T03:54:07.824 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 17/95 2026-02-12T03:54:07.824 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 18/95 2026-02-12T03:54:07.824 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 19/95 2026-02-12T03:54:07.824 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 20/95 2026-02-12T03:54:07.824 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 21/95 2026-02-12T03:54:07.824 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 22/95 2026-02-12T03:54:07.824 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 23/95 2026-02-12T03:54:07.824 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 24/95 2026-02-12T03:54:07.825 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 25/95 2026-02-12T03:54:07.825 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 26/95 2026-02-12T03:54:07.825 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 27/95 2026-02-12T03:54:07.825 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : unzip-6.0-59.el9.x86_64 28/95 2026-02-12T03:54:07.825 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : zip-3.0-35.el9.x86_64 29/95 2026-02-12T03:54:07.825 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 30/95 2026-02-12T03:54:07.825 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 31/95 2026-02-12T03:54:07.825 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 32/95 2026-02-12T03:54:07.825 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 33/95 2026-02-12T03:54:07.825 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : lua-5.4.4-4.el9.x86_64 34/95 2026-02-12T03:54:07.825 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 35/95 2026-02-12T03:54:07.825 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 36/95 2026-02-12T03:54:07.826 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 37/95 2026-02-12T03:54:07.826 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 38/95 2026-02-12T03:54:07.826 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-devel-3.9.25-2.el9.x86_64 39/95 2026-02-12T03:54:07.826 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 40/95 2026-02-12T03:54:07.826 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 41/95 2026-02-12T03:54:07.826 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 42/95 2026-02-12T03:54:07.826 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 43/95 2026-02-12T03:54:07.826 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 44/95 2026-02-12T03:54:07.826 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 45/95 2026-02-12T03:54:07.826 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 46/95 2026-02-12T03:54:07.826 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 47/95 2026-02-12T03:54:07.826 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 48/95 2026-02-12T03:54:07.826 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 49/95 2026-02-12T03:54:07.827 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 50/95 2026-02-12T03:54:07.827 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 51/95 2026-02-12T03:54:07.827 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 52/95 2026-02-12T03:54:07.827 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 53/95 2026-02-12T03:54:07.827 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 54/95 2026-02-12T03:54:07.827 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 55/95 2026-02-12T03:54:07.827 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 56/95 2026-02-12T03:54:07.827 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 57/95 2026-02-12T03:54:07.827 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 58/95 2026-02-12T03:54:07.827 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 59/95 2026-02-12T03:54:07.827 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 60/95 2026-02-12T03:54:07.827 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 61/95 2026-02-12T03:54:07.827 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 62/95 2026-02-12T03:54:07.828 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 63/95 2026-02-12T03:54:07.828 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 64/95 2026-02-12T03:54:07.828 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 65/95 2026-02-12T03:54:07.828 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 66/95 2026-02-12T03:54:07.828 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 67/95 2026-02-12T03:54:07.828 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 68/95 2026-02-12T03:54:07.828 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 69/95 2026-02-12T03:54:07.828 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 70/95 2026-02-12T03:54:07.828 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 71/95 2026-02-12T03:54:07.828 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 72/95 2026-02-12T03:54:07.828 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 73/95 2026-02-12T03:54:07.828 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 74/95 2026-02-12T03:54:07.828 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 75/95 2026-02-12T03:54:07.828 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 76/95 2026-02-12T03:54:07.829 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 77/95 2026-02-12T03:54:07.829 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 78/95 2026-02-12T03:54:07.829 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 79/95 2026-02-12T03:54:07.829 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 80/95 2026-02-12T03:54:07.829 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 81/95 2026-02-12T03:54:07.829 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 82/95 2026-02-12T03:54:07.829 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 83/95 2026-02-12T03:54:07.829 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 84/95 2026-02-12T03:54:07.829 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 85/95 2026-02-12T03:54:07.829 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 86/95 2026-02-12T03:54:07.829 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 87/95 2026-02-12T03:54:07.829 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 88/95 2026-02-12T03:54:07.830 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 89/95 2026-02-12T03:54:07.830 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 90/95 2026-02-12T03:54:07.830 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 91/95 2026-02-12T03:54:07.830 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 92/95 2026-02-12T03:54:07.830 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 93/95 2026-02-12T03:54:07.830 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 94/95 2026-02-12T03:54:07.885 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 95/95 2026-02-12T03:54:07.885 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:07.885 INFO:teuthology.orchestra.run.trial103.stdout:Installed: 2026-02-12T03:54:07.885 INFO:teuthology.orchestra.run.trial103.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-12T03:54:07.885 INFO:teuthology.orchestra.run.trial103.stdout: ceph-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:54:07.886 INFO:teuthology.orchestra.run.trial103.stdout: ceph-grafana-dashboards-2:20.2.0-626.g1279775f.el9.noarch 2026-02-12T03:54:07.886 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mds-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:54:07.886 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mgr-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:54:07.886 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mgr-cephadm-2:20.2.0-626.g1279775f.el9.noarch 2026-02-12T03:54:07.886 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mgr-dashboard-2:20.2.0-626.g1279775f.el9.noarch 2026-02-12T03:54:07.886 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mgr-diskprediction-local-2:20.2.0-626.g1279775f.el9.noarch 2026-02-12T03:54:07.886 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mgr-k8sevents-2:20.2.0-626.g1279775f.el9.noarch 2026-02-12T03:54:07.886 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mgr-modules-core-2:20.2.0-626.g1279775f.el9.noarch 2026-02-12T03:54:07.886 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mgr-rook-2:20.2.0-626.g1279775f.el9.noarch 2026-02-12T03:54:07.886 INFO:teuthology.orchestra.run.trial103.stdout: ceph-mon-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:54:07.886 INFO:teuthology.orchestra.run.trial103.stdout: ceph-osd-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:54:07.886 INFO:teuthology.orchestra.run.trial103.stdout: ceph-prometheus-alerts-2:20.2.0-626.g1279775f.el9.noarch 2026-02-12T03:54:07.887 INFO:teuthology.orchestra.run.trial103.stdout: ceph-volume-2:20.2.0-626.g1279775f.el9.noarch 2026-02-12T03:54:07.887 INFO:teuthology.orchestra.run.trial103.stdout: cephadm-2:20.2.0-626.g1279775f.el9.noarch 2026-02-12T03:54:07.887 INFO:teuthology.orchestra.run.trial103.stdout: cryptsetup-2.8.1-2.el9.x86_64 2026-02-12T03:54:07.887 INFO:teuthology.orchestra.run.trial103.stdout: flexiblas-3.0.4-8.el9.x86_64 2026-02-12T03:54:07.887 INFO:teuthology.orchestra.run.trial103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2026-02-12T03:54:07.887 INFO:teuthology.orchestra.run.trial103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2026-02-12T03:54:07.887 INFO:teuthology.orchestra.run.trial103.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-12T03:54:07.887 INFO:teuthology.orchestra.run.trial103.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-12T03:54:07.887 INFO:teuthology.orchestra.run.trial103.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-12T03:54:07.887 INFO:teuthology.orchestra.run.trial103.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-12T03:54:07.887 INFO:teuthology.orchestra.run.trial103.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-12T03:54:07.887 INFO:teuthology.orchestra.run.trial103.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-12T03:54:07.888 INFO:teuthology.orchestra.run.trial103.stdout: lua-5.4.4-4.el9.x86_64 2026-02-12T03:54:07.888 INFO:teuthology.orchestra.run.trial103.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-12T03:54:07.888 INFO:teuthology.orchestra.run.trial103.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-12T03:54:07.888 INFO:teuthology.orchestra.run.trial103.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-12T03:54:07.888 INFO:teuthology.orchestra.run.trial103.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-12T03:54:07.888 INFO:teuthology.orchestra.run.trial103.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-12T03:54:07.888 INFO:teuthology.orchestra.run.trial103.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-12T03:54:07.888 INFO:teuthology.orchestra.run.trial103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-12T03:54:07.888 INFO:teuthology.orchestra.run.trial103.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-12T03:54:07.888 INFO:teuthology.orchestra.run.trial103.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-12T03:54:07.888 INFO:teuthology.orchestra.run.trial103.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-12T03:54:07.888 INFO:teuthology.orchestra.run.trial103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-12T03:54:07.889 INFO:teuthology.orchestra.run.trial103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-12T03:54:07.889 INFO:teuthology.orchestra.run.trial103.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-12T03:54:07.889 INFO:teuthology.orchestra.run.trial103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-12T03:54:07.889 INFO:teuthology.orchestra.run.trial103.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-12T03:54:07.889 INFO:teuthology.orchestra.run.trial103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-12T03:54:07.889 INFO:teuthology.orchestra.run.trial103.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-12T03:54:07.889 INFO:teuthology.orchestra.run.trial103.stdout: python3-devel-3.9.25-2.el9.x86_64 2026-02-12T03:54:07.889 INFO:teuthology.orchestra.run.trial103.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-12T03:54:07.889 INFO:teuthology.orchestra.run.trial103.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-12T03:54:07.889 INFO:teuthology.orchestra.run.trial103.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-12T03:54:07.889 INFO:teuthology.orchestra.run.trial103.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-12T03:54:07.889 INFO:teuthology.orchestra.run.trial103.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-12T03:54:07.889 INFO:teuthology.orchestra.run.trial103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-12T03:54:07.890 INFO:teuthology.orchestra.run.trial103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-12T03:54:07.890 INFO:teuthology.orchestra.run.trial103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-12T03:54:07.890 INFO:teuthology.orchestra.run.trial103.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-12T03:54:07.890 INFO:teuthology.orchestra.run.trial103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-12T03:54:07.890 INFO:teuthology.orchestra.run.trial103.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-12T03:54:07.890 INFO:teuthology.orchestra.run.trial103.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-12T03:54:07.890 INFO:teuthology.orchestra.run.trial103.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-12T03:54:07.890 INFO:teuthology.orchestra.run.trial103.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-12T03:54:07.890 INFO:teuthology.orchestra.run.trial103.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-12T03:54:07.890 INFO:teuthology.orchestra.run.trial103.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-12T03:54:07.890 INFO:teuthology.orchestra.run.trial103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-12T03:54:07.890 INFO:teuthology.orchestra.run.trial103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-12T03:54:07.890 INFO:teuthology.orchestra.run.trial103.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-12T03:54:07.891 INFO:teuthology.orchestra.run.trial103.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-12T03:54:07.891 INFO:teuthology.orchestra.run.trial103.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-12T03:54:07.891 INFO:teuthology.orchestra.run.trial103.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-12T03:54:07.891 INFO:teuthology.orchestra.run.trial103.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-12T03:54:07.891 INFO:teuthology.orchestra.run.trial103.stdout: python3-ply-3.11-14.el9.noarch 2026-02-12T03:54:07.891 INFO:teuthology.orchestra.run.trial103.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-12T03:54:07.891 INFO:teuthology.orchestra.run.trial103.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-12T03:54:07.891 INFO:teuthology.orchestra.run.trial103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-12T03:54:07.891 INFO:teuthology.orchestra.run.trial103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-12T03:54:07.891 INFO:teuthology.orchestra.run.trial103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-12T03:54:07.891 INFO:teuthology.orchestra.run.trial103.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-12T03:54:07.891 INFO:teuthology.orchestra.run.trial103.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-12T03:54:07.891 INFO:teuthology.orchestra.run.trial103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-12T03:54:07.892 INFO:teuthology.orchestra.run.trial103.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-12T03:54:07.892 INFO:teuthology.orchestra.run.trial103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-12T03:54:07.892 INFO:teuthology.orchestra.run.trial103.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-12T03:54:07.892 INFO:teuthology.orchestra.run.trial103.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-12T03:54:07.892 INFO:teuthology.orchestra.run.trial103.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-12T03:54:07.892 INFO:teuthology.orchestra.run.trial103.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-12T03:54:07.892 INFO:teuthology.orchestra.run.trial103.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-12T03:54:07.892 INFO:teuthology.orchestra.run.trial103.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-12T03:54:07.892 INFO:teuthology.orchestra.run.trial103.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-12T03:54:07.892 INFO:teuthology.orchestra.run.trial103.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-12T03:54:07.892 INFO:teuthology.orchestra.run.trial103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-12T03:54:07.892 INFO:teuthology.orchestra.run.trial103.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-12T03:54:07.892 INFO:teuthology.orchestra.run.trial103.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-12T03:54:07.893 INFO:teuthology.orchestra.run.trial103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-12T03:54:07.893 INFO:teuthology.orchestra.run.trial103.stdout: unzip-6.0-59.el9.x86_64 2026-02-12T03:54:07.893 INFO:teuthology.orchestra.run.trial103.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-12T03:54:07.893 INFO:teuthology.orchestra.run.trial103.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-12T03:54:07.893 INFO:teuthology.orchestra.run.trial103.stdout: zip-3.0-35.el9.x86_64 2026-02-12T03:54:07.893 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:07.893 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:07.940 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install ceph-base 2026-02-12T03:54:08.245 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:37 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:08.310 INFO:teuthology.orchestra.run.trial103.stdout:Package ceph-base-2:20.2.0-626.g1279775f.el9.x86_64 is already installed. 2026-02-12T03:54:08.328 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:08.329 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:08.329 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:08.350 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install cephadm 2026-02-12T03:54:08.652 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:37 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:08.718 INFO:teuthology.orchestra.run.trial103.stdout:Package cephadm-2:20.2.0-626.g1279775f.el9.noarch is already installed. 2026-02-12T03:54:08.737 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:08.737 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:08.737 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:08.759 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install ceph-immutable-object-cache 2026-02-12T03:54:09.060 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:38 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:09.144 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:09.144 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:09.144 INFO:teuthology.orchestra.run.trial103.stdout: Package Arch Version Repo Size 2026-02-12T03:54:09.144 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:09.144 INFO:teuthology.orchestra.run.trial103.stdout:Installing: 2026-02-12T03:54:09.144 INFO:teuthology.orchestra.run.trial103.stdout: ceph-immutable-object-cache x86_64 2:20.2.0-626.g1279775f.el9 ceph 154 k 2026-02-12T03:54:09.144 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:09.145 INFO:teuthology.orchestra.run.trial103.stdout:Transaction Summary 2026-02-12T03:54:09.145 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:09.145 INFO:teuthology.orchestra.run.trial103.stdout:Install 1 Package 2026-02-12T03:54:09.145 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:09.145 INFO:teuthology.orchestra.run.trial103.stdout:Total download size: 154 k 2026-02-12T03:54:09.145 INFO:teuthology.orchestra.run.trial103.stdout:Installed size: 447 k 2026-02-12T03:54:09.145 INFO:teuthology.orchestra.run.trial103.stdout:Downloading Packages: 2026-02-12T03:54:09.516 INFO:teuthology.orchestra.run.trial103.stdout:ceph-immutable-object-cache-20.2.0-626.g1279775 415 kB/s | 154 kB 00:00 2026-02-12T03:54:09.516 INFO:teuthology.orchestra.run.trial103.stdout:-------------------------------------------------------------------------------- 2026-02-12T03:54:09.516 INFO:teuthology.orchestra.run.trial103.stdout:Total 414 kB/s | 154 kB 00:00 2026-02-12T03:54:09.516 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction check 2026-02-12T03:54:09.520 INFO:teuthology.orchestra.run.trial103.stdout:Transaction check succeeded. 2026-02-12T03:54:09.520 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction test 2026-02-12T03:54:09.554 INFO:teuthology.orchestra.run.trial103.stdout:Transaction test succeeded. 2026-02-12T03:54:09.554 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction 2026-02-12T03:54:09.611 INFO:teuthology.orchestra.run.trial103.stdout: Preparing : 1/1 2026-02-12T03:54:09.613 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-immutable-object-cache-2:20.2.0-626.g1279775f.e 1/1 2026-02-12T03:54:09.635 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.2.0-626.g1279775f.e 1/1 2026-02-12T03:54:09.635 INFO:teuthology.orchestra.run.trial103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T03:54:09.635 INFO:teuthology.orchestra.run.trial103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-12T03:54:09.635 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:09.914 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-immutable-object-cache-2:20.2.0-626.g1279775f.e 1/1 2026-02-12T03:54:09.915 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:09.915 INFO:teuthology.orchestra.run.trial103.stdout:Installed: 2026-02-12T03:54:09.915 INFO:teuthology.orchestra.run.trial103.stdout: ceph-immutable-object-cache-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:54:09.915 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:09.915 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:09.942 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install ceph-mgr 2026-02-12T03:54:10.242 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:39 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:10.307 INFO:teuthology.orchestra.run.trial103.stdout:Package ceph-mgr-2:20.2.0-626.g1279775f.el9.x86_64 is already installed. 2026-02-12T03:54:10.325 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:10.326 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:10.326 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:10.347 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install ceph-mgr-dashboard 2026-02-12T03:54:10.650 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:39 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:10.715 INFO:teuthology.orchestra.run.trial103.stdout:Package ceph-mgr-dashboard-2:20.2.0-626.g1279775f.el9.noarch is already installed. 2026-02-12T03:54:10.733 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:10.734 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:10.734 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:10.755 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-12T03:54:11.054 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:40 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:11.119 INFO:teuthology.orchestra.run.trial103.stdout:Package ceph-mgr-diskprediction-local-2:20.2.0-626.g1279775f.el9.noarch is already installed. 2026-02-12T03:54:11.137 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:11.137 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:11.137 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:11.159 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install ceph-mgr-rook 2026-02-12T03:54:11.462 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:40 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:11.526 INFO:teuthology.orchestra.run.trial103.stdout:Package ceph-mgr-rook-2:20.2.0-626.g1279775f.el9.noarch is already installed. 2026-02-12T03:54:11.545 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:11.545 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:11.545 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:11.567 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install ceph-mgr-cephadm 2026-02-12T03:54:11.864 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:40 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:11.928 INFO:teuthology.orchestra.run.trial103.stdout:Package ceph-mgr-cephadm-2:20.2.0-626.g1279775f.el9.noarch is already installed. 2026-02-12T03:54:11.947 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:11.947 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:11.947 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:11.969 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install ceph-fuse 2026-02-12T03:54:12.268 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:41 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:12.352 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:12.352 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:12.352 INFO:teuthology.orchestra.run.trial103.stdout: Package Architecture Version Repository Size 2026-02-12T03:54:12.352 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:12.353 INFO:teuthology.orchestra.run.trial103.stdout:Installing: 2026-02-12T03:54:12.353 INFO:teuthology.orchestra.run.trial103.stdout: ceph-fuse x86_64 2:20.2.0-626.g1279775f.el9 ceph 943 k 2026-02-12T03:54:12.353 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:12.353 INFO:teuthology.orchestra.run.trial103.stdout:Transaction Summary 2026-02-12T03:54:12.353 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:12.353 INFO:teuthology.orchestra.run.trial103.stdout:Install 1 Package 2026-02-12T03:54:12.353 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:12.353 INFO:teuthology.orchestra.run.trial103.stdout:Total download size: 943 k 2026-02-12T03:54:12.353 INFO:teuthology.orchestra.run.trial103.stdout:Installed size: 2.7 M 2026-02-12T03:54:12.353 INFO:teuthology.orchestra.run.trial103.stdout:Downloading Packages: 2026-02-12T03:54:13.036 INFO:teuthology.orchestra.run.trial103.stdout:ceph-fuse-20.2.0-626.g1279775f.el9.x86_64.rpm 1.3 MB/s | 943 kB 00:00 2026-02-12T03:54:13.037 INFO:teuthology.orchestra.run.trial103.stdout:-------------------------------------------------------------------------------- 2026-02-12T03:54:13.037 INFO:teuthology.orchestra.run.trial103.stdout:Total 1.3 MB/s | 943 kB 00:00 2026-02-12T03:54:13.037 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction check 2026-02-12T03:54:13.042 INFO:teuthology.orchestra.run.trial103.stdout:Transaction check succeeded. 2026-02-12T03:54:13.042 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction test 2026-02-12T03:54:13.077 INFO:teuthology.orchestra.run.trial103.stdout:Transaction test succeeded. 2026-02-12T03:54:13.077 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction 2026-02-12T03:54:13.166 INFO:teuthology.orchestra.run.trial103.stdout: Preparing : 1/1 2026-02-12T03:54:13.183 INFO:teuthology.orchestra.run.trial103.stdout: Installing : ceph-fuse-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:13.389 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: ceph-fuse-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:13.446 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : ceph-fuse-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:13.447 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:13.447 INFO:teuthology.orchestra.run.trial103.stdout:Installed: 2026-02-12T03:54:13.447 INFO:teuthology.orchestra.run.trial103.stdout: ceph-fuse-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:54:13.447 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:13.447 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:13.475 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install ceph-volume 2026-02-12T03:54:13.779 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:42 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:13.844 INFO:teuthology.orchestra.run.trial103.stdout:Package ceph-volume-2:20.2.0-626.g1279775f.el9.noarch is already installed. 2026-02-12T03:54:13.863 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:13.863 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:13.863 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:13.885 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install librados-devel 2026-02-12T03:54:14.188 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:43 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:14.272 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:14.272 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:14.272 INFO:teuthology.orchestra.run.trial103.stdout: Package Arch Version Repo Size 2026-02-12T03:54:14.272 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:14.272 INFO:teuthology.orchestra.run.trial103.stdout:Installing: 2026-02-12T03:54:14.272 INFO:teuthology.orchestra.run.trial103.stdout: librados-devel x86_64 2:20.2.0-626.g1279775f.el9 ceph 126 k 2026-02-12T03:54:14.272 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:14.273 INFO:teuthology.orchestra.run.trial103.stdout:Transaction Summary 2026-02-12T03:54:14.273 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:14.273 INFO:teuthology.orchestra.run.trial103.stdout:Install 1 Package 2026-02-12T03:54:14.273 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:14.273 INFO:teuthology.orchestra.run.trial103.stdout:Total download size: 126 k 2026-02-12T03:54:14.273 INFO:teuthology.orchestra.run.trial103.stdout:Installed size: 449 k 2026-02-12T03:54:14.273 INFO:teuthology.orchestra.run.trial103.stdout:Downloading Packages: 2026-02-12T03:54:14.573 INFO:teuthology.orchestra.run.trial103.stdout:librados-devel-20.2.0-626.g1279775f.el9.x86_64. 419 kB/s | 126 kB 00:00 2026-02-12T03:54:14.574 INFO:teuthology.orchestra.run.trial103.stdout:-------------------------------------------------------------------------------- 2026-02-12T03:54:14.574 INFO:teuthology.orchestra.run.trial103.stdout:Total 417 kB/s | 126 kB 00:00 2026-02-12T03:54:14.574 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction check 2026-02-12T03:54:14.576 INFO:teuthology.orchestra.run.trial103.stdout:Transaction check succeeded. 2026-02-12T03:54:14.577 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction test 2026-02-12T03:54:14.609 INFO:teuthology.orchestra.run.trial103.stdout:Transaction test succeeded. 2026-02-12T03:54:14.609 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction 2026-02-12T03:54:14.663 INFO:teuthology.orchestra.run.trial103.stdout: Preparing : 1/1 2026-02-12T03:54:14.679 INFO:teuthology.orchestra.run.trial103.stdout: Installing : librados-devel-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:14.761 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: librados-devel-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:14.817 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : librados-devel-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:14.818 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:14.818 INFO:teuthology.orchestra.run.trial103.stdout:Installed: 2026-02-12T03:54:14.818 INFO:teuthology.orchestra.run.trial103.stdout: librados-devel-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:54:14.818 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:14.818 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:14.846 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install libcephfs2 2026-02-12T03:54:15.151 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:44 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:15.216 INFO:teuthology.orchestra.run.trial103.stdout:Package libcephfs2-2:20.2.0-626.g1279775f.el9.x86_64 is already installed. 2026-02-12T03:54:15.234 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:15.235 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:15.235 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:15.257 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install libcephfs-devel 2026-02-12T03:54:15.554 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:44 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:15.638 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:15.638 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:15.638 INFO:teuthology.orchestra.run.trial103.stdout: Package Arch Version Repo Size 2026-02-12T03:54:15.638 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:15.638 INFO:teuthology.orchestra.run.trial103.stdout:Installing: 2026-02-12T03:54:15.638 INFO:teuthology.orchestra.run.trial103.stdout: libcephfs-devel x86_64 2:20.2.0-626.g1279775f.el9 ceph 34 k 2026-02-12T03:54:15.639 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:15.639 INFO:teuthology.orchestra.run.trial103.stdout:Transaction Summary 2026-02-12T03:54:15.639 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:15.639 INFO:teuthology.orchestra.run.trial103.stdout:Install 1 Package 2026-02-12T03:54:15.639 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:15.639 INFO:teuthology.orchestra.run.trial103.stdout:Total download size: 34 k 2026-02-12T03:54:15.639 INFO:teuthology.orchestra.run.trial103.stdout:Installed size: 155 k 2026-02-12T03:54:15.639 INFO:teuthology.orchestra.run.trial103.stdout:Downloading Packages: 2026-02-12T03:54:15.740 INFO:teuthology.orchestra.run.trial103.stdout:libcephfs-devel-20.2.0-626.g1279775f.el9.x86_64 338 kB/s | 34 kB 00:00 2026-02-12T03:54:15.741 INFO:teuthology.orchestra.run.trial103.stdout:-------------------------------------------------------------------------------- 2026-02-12T03:54:15.741 INFO:teuthology.orchestra.run.trial103.stdout:Total 335 kB/s | 34 kB 00:00 2026-02-12T03:54:15.741 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction check 2026-02-12T03:54:15.743 INFO:teuthology.orchestra.run.trial103.stdout:Transaction check succeeded. 2026-02-12T03:54:15.743 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction test 2026-02-12T03:54:15.745 INFO:teuthology.orchestra.run.trial103.stdout:Transaction test succeeded. 2026-02-12T03:54:15.745 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction 2026-02-12T03:54:15.766 INFO:teuthology.orchestra.run.trial103.stdout: Preparing : 1/1 2026-02-12T03:54:15.782 INFO:teuthology.orchestra.run.trial103.stdout: Installing : libcephfs-devel-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:15.836 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: libcephfs-devel-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:15.892 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : libcephfs-devel-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:15.892 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:15.892 INFO:teuthology.orchestra.run.trial103.stdout:Installed: 2026-02-12T03:54:15.893 INFO:teuthology.orchestra.run.trial103.stdout: libcephfs-devel-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:54:15.893 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:15.893 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:15.920 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install librados2 2026-02-12T03:54:16.224 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:45 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:16.289 INFO:teuthology.orchestra.run.trial103.stdout:Package librados2-2:20.2.0-626.g1279775f.el9.x86_64 is already installed. 2026-02-12T03:54:16.307 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:16.308 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:16.308 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:16.330 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install librbd1 2026-02-12T03:54:16.633 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:45 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:16.697 INFO:teuthology.orchestra.run.trial103.stdout:Package librbd1-2:20.2.0-626.g1279775f.el9.x86_64 is already installed. 2026-02-12T03:54:16.716 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:16.716 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:16.717 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:16.738 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install python3-rados 2026-02-12T03:54:17.043 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:46 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:17.110 INFO:teuthology.orchestra.run.trial103.stdout:Package python3-rados-2:20.2.0-626.g1279775f.el9.x86_64 is already installed. 2026-02-12T03:54:17.128 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:17.128 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:17.128 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:17.151 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install python3-rgw 2026-02-12T03:54:17.457 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:46 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:17.522 INFO:teuthology.orchestra.run.trial103.stdout:Package python3-rgw-2:20.2.0-626.g1279775f.el9.x86_64 is already installed. 2026-02-12T03:54:17.540 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:17.540 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:17.541 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:17.562 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install python3-cephfs 2026-02-12T03:54:17.861 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:46 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:17.926 INFO:teuthology.orchestra.run.trial103.stdout:Package python3-cephfs-2:20.2.0-626.g1279775f.el9.x86_64 is already installed. 2026-02-12T03:54:17.944 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:17.945 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:17.945 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:17.966 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install python3-rbd 2026-02-12T03:54:18.269 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:47 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:18.335 INFO:teuthology.orchestra.run.trial103.stdout:Package python3-rbd-2:20.2.0-626.g1279775f.el9.x86_64 is already installed. 2026-02-12T03:54:18.354 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:18.354 INFO:teuthology.orchestra.run.trial103.stdout:Nothing to do. 2026-02-12T03:54:18.355 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:18.377 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install rbd-fuse 2026-02-12T03:54:18.682 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:47 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:18.765 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:18.766 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:18.766 INFO:teuthology.orchestra.run.trial103.stdout: Package Architecture Version Repository Size 2026-02-12T03:54:18.766 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:18.766 INFO:teuthology.orchestra.run.trial103.stdout:Installing: 2026-02-12T03:54:18.766 INFO:teuthology.orchestra.run.trial103.stdout: rbd-fuse x86_64 2:20.2.0-626.g1279775f.el9 ceph 91 k 2026-02-12T03:54:18.766 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:18.766 INFO:teuthology.orchestra.run.trial103.stdout:Transaction Summary 2026-02-12T03:54:18.766 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:18.767 INFO:teuthology.orchestra.run.trial103.stdout:Install 1 Package 2026-02-12T03:54:18.767 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:18.767 INFO:teuthology.orchestra.run.trial103.stdout:Total download size: 91 k 2026-02-12T03:54:18.767 INFO:teuthology.orchestra.run.trial103.stdout:Installed size: 238 k 2026-02-12T03:54:18.767 INFO:teuthology.orchestra.run.trial103.stdout:Downloading Packages: 2026-02-12T03:54:18.925 INFO:teuthology.orchestra.run.trial103.stdout:rbd-fuse-20.2.0-626.g1279775f.el9.x86_64.rpm 575 kB/s | 91 kB 00:00 2026-02-12T03:54:18.925 INFO:teuthology.orchestra.run.trial103.stdout:-------------------------------------------------------------------------------- 2026-02-12T03:54:18.925 INFO:teuthology.orchestra.run.trial103.stdout:Total 572 kB/s | 91 kB 00:00 2026-02-12T03:54:18.925 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction check 2026-02-12T03:54:18.929 INFO:teuthology.orchestra.run.trial103.stdout:Transaction check succeeded. 2026-02-12T03:54:18.930 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction test 2026-02-12T03:54:18.962 INFO:teuthology.orchestra.run.trial103.stdout:Transaction test succeeded. 2026-02-12T03:54:18.962 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction 2026-02-12T03:54:19.013 INFO:teuthology.orchestra.run.trial103.stdout: Preparing : 1/1 2026-02-12T03:54:19.029 INFO:teuthology.orchestra.run.trial103.stdout: Installing : rbd-fuse-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:19.110 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: rbd-fuse-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:19.168 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : rbd-fuse-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:19.168 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:19.168 INFO:teuthology.orchestra.run.trial103.stdout:Installed: 2026-02-12T03:54:19.169 INFO:teuthology.orchestra.run.trial103.stdout: rbd-fuse-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:54:19.169 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:19.169 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:19.198 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install rbd-mirror 2026-02-12T03:54:19.510 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:48 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:19.594 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:19.595 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:19.595 INFO:teuthology.orchestra.run.trial103.stdout: Package Arch Version Repo Size 2026-02-12T03:54:19.595 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:19.595 INFO:teuthology.orchestra.run.trial103.stdout:Installing: 2026-02-12T03:54:19.595 INFO:teuthology.orchestra.run.trial103.stdout: rbd-mirror x86_64 2:20.2.0-626.g1279775f.el9 ceph 2.9 M 2026-02-12T03:54:19.595 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:19.595 INFO:teuthology.orchestra.run.trial103.stdout:Transaction Summary 2026-02-12T03:54:19.595 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:19.596 INFO:teuthology.orchestra.run.trial103.stdout:Install 1 Package 2026-02-12T03:54:19.596 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:19.596 INFO:teuthology.orchestra.run.trial103.stdout:Total download size: 2.9 M 2026-02-12T03:54:19.596 INFO:teuthology.orchestra.run.trial103.stdout:Installed size: 11 M 2026-02-12T03:54:19.596 INFO:teuthology.orchestra.run.trial103.stdout:Downloading Packages: 2026-02-12T03:54:20.183 INFO:teuthology.orchestra.run.trial103.stdout:rbd-mirror-20.2.0-626.g1279775f.el9.x86_64.rpm 5.0 MB/s | 2.9 MB 00:00 2026-02-12T03:54:20.183 INFO:teuthology.orchestra.run.trial103.stdout:-------------------------------------------------------------------------------- 2026-02-12T03:54:20.183 INFO:teuthology.orchestra.run.trial103.stdout:Total 5.0 MB/s | 2.9 MB 00:00 2026-02-12T03:54:20.184 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction check 2026-02-12T03:54:20.188 INFO:teuthology.orchestra.run.trial103.stdout:Transaction check succeeded. 2026-02-12T03:54:20.188 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction test 2026-02-12T03:54:20.228 INFO:teuthology.orchestra.run.trial103.stdout:Transaction test succeeded. 2026-02-12T03:54:20.228 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction 2026-02-12T03:54:20.413 INFO:teuthology.orchestra.run.trial103.stdout: Preparing : 1/1 2026-02-12T03:54:20.414 INFO:teuthology.orchestra.run.trial103.stdout: Installing : rbd-mirror-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:20.436 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: rbd-mirror-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:20.436 INFO:teuthology.orchestra.run.trial103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T03:54:20.437 INFO:teuthology.orchestra.run.trial103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-12T03:54:20.437 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-12T03:54:20.437 INFO:teuthology.orchestra.run.trial103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-12T03:54:20.437 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:20.728 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : rbd-mirror-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:20.729 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:20.729 INFO:teuthology.orchestra.run.trial103.stdout:Installed: 2026-02-12T03:54:20.729 INFO:teuthology.orchestra.run.trial103.stdout: rbd-mirror-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:54:20.729 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:20.729 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:20.757 DEBUG:teuthology.orchestra.run.trial103:> sudo yum -y install rbd-nbd 2026-02-12T03:54:21.061 INFO:teuthology.orchestra.run.trial103.stdout:Last metadata expiration check: 0:00:50 ago on Thu 12 Feb 2026 03:53:31 AM UTC. 2026-02-12T03:54:21.144 INFO:teuthology.orchestra.run.trial103.stdout:Dependencies resolved. 2026-02-12T03:54:21.145 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:21.145 INFO:teuthology.orchestra.run.trial103.stdout: Package Architecture Version Repository Size 2026-02-12T03:54:21.145 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:21.145 INFO:teuthology.orchestra.run.trial103.stdout:Installing: 2026-02-12T03:54:21.145 INFO:teuthology.orchestra.run.trial103.stdout: rbd-nbd x86_64 2:20.2.0-626.g1279775f.el9 ceph 180 k 2026-02-12T03:54:21.145 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:21.145 INFO:teuthology.orchestra.run.trial103.stdout:Transaction Summary 2026-02-12T03:54:21.145 INFO:teuthology.orchestra.run.trial103.stdout:================================================================================ 2026-02-12T03:54:21.145 INFO:teuthology.orchestra.run.trial103.stdout:Install 1 Package 2026-02-12T03:54:21.146 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:21.146 INFO:teuthology.orchestra.run.trial103.stdout:Total download size: 180 k 2026-02-12T03:54:21.146 INFO:teuthology.orchestra.run.trial103.stdout:Installed size: 498 k 2026-02-12T03:54:21.146 INFO:teuthology.orchestra.run.trial103.stdout:Downloading Packages: 2026-02-12T03:54:21.245 INFO:teuthology.orchestra.run.trial103.stdout:rbd-nbd-20.2.0-626.g1279775f.el9.x86_64.rpm 1.8 MB/s | 180 kB 00:00 2026-02-12T03:54:21.245 INFO:teuthology.orchestra.run.trial103.stdout:-------------------------------------------------------------------------------- 2026-02-12T03:54:21.245 INFO:teuthology.orchestra.run.trial103.stdout:Total 1.8 MB/s | 180 kB 00:00 2026-02-12T03:54:21.245 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction check 2026-02-12T03:54:21.250 INFO:teuthology.orchestra.run.trial103.stdout:Transaction check succeeded. 2026-02-12T03:54:21.250 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction test 2026-02-12T03:54:21.282 INFO:teuthology.orchestra.run.trial103.stdout:Transaction test succeeded. 2026-02-12T03:54:21.282 INFO:teuthology.orchestra.run.trial103.stdout:Running transaction 2026-02-12T03:54:21.337 INFO:teuthology.orchestra.run.trial103.stdout: Preparing : 1/1 2026-02-12T03:54:21.354 INFO:teuthology.orchestra.run.trial103.stdout: Installing : rbd-nbd-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:21.435 INFO:teuthology.orchestra.run.trial103.stdout: Running scriptlet: rbd-nbd-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:21.493 INFO:teuthology.orchestra.run.trial103.stdout: Verifying : rbd-nbd-2:20.2.0-626.g1279775f.el9.x86_64 1/1 2026-02-12T03:54:21.493 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:21.493 INFO:teuthology.orchestra.run.trial103.stdout:Installed: 2026-02-12T03:54:21.493 INFO:teuthology.orchestra.run.trial103.stdout: rbd-nbd-2:20.2.0-626.g1279775f.el9.x86_64 2026-02-12T03:54:21.493 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:21.493 INFO:teuthology.orchestra.run.trial103.stdout:Complete! 2026-02-12T03:54:21.524 DEBUG:teuthology.parallel:result is None 2026-02-12T03:54:21.524 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1279775fef08a9169052c6dd6a10581353958aac 2026-02-12T03:54:21.688 DEBUG:teuthology.orchestra.run.trial103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-12T03:54:21.709 INFO:teuthology.orchestra.run.trial103.stdout:20.2.0-626.g1279775f.el9 2026-02-12T03:54:21.710 INFO:teuthology.packaging:The installed version of ceph is 20.2.0-626.g1279775f.el9 2026-02-12T03:54:21.710 INFO:teuthology.task.install:The correct ceph version 20.2.0-626.g1279775f is installed. 2026-02-12T03:54:21.713 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-12T03:54:21.713 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:21.713 DEBUG:teuthology.orchestra.run.trial103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-12T03:54:21.777 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-12T03:54:21.777 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:21.777 DEBUG:teuthology.orchestra.run.trial103:> sudo dd of=/usr/bin/daemon-helper 2026-02-12T03:54:21.842 DEBUG:teuthology.orchestra.run.trial103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-12T03:54:21.906 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-12T03:54:21.906 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:21.907 DEBUG:teuthology.orchestra.run.trial103:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-12T03:54:21.971 DEBUG:teuthology.orchestra.run.trial103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-12T03:54:22.036 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-12T03:54:22.036 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:22.036 DEBUG:teuthology.orchestra.run.trial103:> sudo dd of=/usr/bin/stdin-killer 2026-02-12T03:54:22.099 DEBUG:teuthology.orchestra.run.trial103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-12T03:54:22.164 INFO:teuthology.run_tasks:Running task ceph... 2026-02-12T03:54:22.280 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-12T03:54:22.280 DEBUG:teuthology.orchestra.run.trial103:> sudo chmod 777 /var/log/ceph 2026-02-12T03:54:22.305 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-12T03:54:22.305 DEBUG:teuthology.orchestra.run.trial103:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-12T03:54:22.367 INFO:tasks.ceph:Creating extra log directories... 2026-02-12T03:54:22.367 DEBUG:teuthology.orchestra.run.trial103:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-12T03:54:22.439 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-12T03:54:22.439 INFO:tasks.ceph:config {'conf': {'osd': {'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'btree', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore onode segment size': '1024K', 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 1, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['Error', 'overall HEALTH_', '\\(OBJECT_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_NEARFULL\\)', '\\(SLOW_OPS\\)', '\\(TOO_FEW_PGS\\)', 'Monitor daemon marked osd\\.[[:digit:]]+ down, but it is still running', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-12T03:54:22.439 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-11_20:27:55-rados-wip-yuri3-testing-2026-02-09-1550-tentacle-distro-default-trial/45778', 'branch': 'wip-yuri3-testing-2026-02-09-1550-tentacle', 'description': 'rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/many msgr/async-v2only objectstore/{bluestore/{alloc$/{btree} base mem$/{low} onode-segment$/{1M} write$/{v2/{compr$/{no$/{no}} v2}}}} rados supported-random-distro$/{centos_latest}}', 'email': None, 'first_in_suite': False, 'flavor': 'default', 'job_id': '45778', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'yuriw-2026-02-11_20:27:55-rados-wip-yuri3-testing-2026-02-09-1550-tentacle-distro-default-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2026-02-09-1550-tentacle'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 1, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'btree', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore onode segment size': '1024K', 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': '1279775fef08a9169052c6dd6a10581353958aac'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1279775fef08a9169052c6dd6a10581353958aac'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri3-testing-2026-02-09-1550-tentacle', 'sha1': '1279775fef08a9169052c6dd6a10581353958aac'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 4009, 'sha1': '1279775fef08a9169052c6dd6a10581353958aac', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri3-testing-2026-02-09-1550-tentacle', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_1279775fef08a9169052c6dd6a10581353958aac/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '1279775fef08a9169052c6dd6a10581353958aac', 'targets': {'trial103.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIKSZYnefBHqnB+Zs4qYd7szXoh3dS315Y2eEMDVeUg71wl4BSpqk09Thxt77eUFfdHRY/KX8z744A/dV9l47ww='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'btree', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore onode segment size': '1024K', 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 1, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'create_rbd_pool': False, 'log-ignorelist': ['Error', 'overall HEALTH_', '\\(OBJECT_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_NEARFULL\\)', '\\(SLOW_OPS\\)', '\\(TOO_FEW_PGS\\)', 'Monitor daemon marked osd\\.[[:digit:]]+ down, but it is still running', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force'], 'flavor': 'default', 'fs': 'xfs', 'sha1': '1279775fef08a9169052c6dd6a10581353958aac', 'cluster': 'ceph'}}, {'backfill_toofull': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'reimage-ssh-down', 'teuthology_sha1': '5f66ecfb34c0370410e78b3ee641753d19da653b', 'timestamp': '2026-02-11_20:27:55', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.2161924'} 2026-02-12T03:54:22.440 DEBUG:teuthology.orchestra.run.trial103:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-12T03:54:22.486 DEBUG:teuthology.orchestra.run.trial103:> sudo install -d -m0777 -- /var/run/ceph 2026-02-12T03:54:22.551 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:22.551 DEBUG:teuthology.orchestra.run.trial103:> dd if=/scratch_devs of=/dev/stdout 2026-02-12T03:54:22.608 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-12T03:54:22.608 DEBUG:teuthology.orchestra.run.trial103:> stat /dev/vg_nvme/lv_1 2026-02-12T03:54:22.664 INFO:teuthology.orchestra.run.trial103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-12T03:54:22.664 INFO:teuthology.orchestra.run.trial103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T03:54:22.664 INFO:teuthology.orchestra.run.trial103.stdout:Device: 6h/6d Inode: 1302 Links: 1 2026-02-12T03:54:22.664 INFO:teuthology.orchestra.run.trial103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T03:54:22.665 INFO:teuthology.orchestra.run.trial103.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T03:54:22.665 INFO:teuthology.orchestra.run.trial103.stdout:Access: 2026-02-12 03:54:20.575199991 +0000 2026-02-12T03:54:22.665 INFO:teuthology.orchestra.run.trial103.stdout:Modify: 2026-02-12 03:53:40.167893922 +0000 2026-02-12T03:54:22.665 INFO:teuthology.orchestra.run.trial103.stdout:Change: 2026-02-12 03:53:40.167893922 +0000 2026-02-12T03:54:22.665 INFO:teuthology.orchestra.run.trial103.stdout: Birth: 2026-02-12 03:53:40.167893922 +0000 2026-02-12T03:54:22.665 DEBUG:teuthology.orchestra.run.trial103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-12T03:54:22.726 INFO:teuthology.orchestra.run.trial103.stderr:1+0 records in 2026-02-12T03:54:22.727 INFO:teuthology.orchestra.run.trial103.stderr:1+0 records out 2026-02-12T03:54:22.727 INFO:teuthology.orchestra.run.trial103.stderr:512 bytes copied, 0.000137884 s, 3.7 MB/s 2026-02-12T03:54:22.728 DEBUG:teuthology.orchestra.run.trial103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-12T03:54:22.785 DEBUG:teuthology.orchestra.run.trial103:> stat /dev/vg_nvme/lv_2 2026-02-12T03:54:22.841 INFO:teuthology.orchestra.run.trial103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-12T03:54:22.841 INFO:teuthology.orchestra.run.trial103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T03:54:22.842 INFO:teuthology.orchestra.run.trial103.stdout:Device: 6h/6d Inode: 1306 Links: 1 2026-02-12T03:54:22.842 INFO:teuthology.orchestra.run.trial103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T03:54:22.842 INFO:teuthology.orchestra.run.trial103.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T03:54:22.842 INFO:teuthology.orchestra.run.trial103.stdout:Access: 2026-02-12 03:54:20.575199991 +0000 2026-02-12T03:54:22.842 INFO:teuthology.orchestra.run.trial103.stdout:Modify: 2026-02-12 03:53:40.167893922 +0000 2026-02-12T03:54:22.842 INFO:teuthology.orchestra.run.trial103.stdout:Change: 2026-02-12 03:53:40.167893922 +0000 2026-02-12T03:54:22.842 INFO:teuthology.orchestra.run.trial103.stdout: Birth: 2026-02-12 03:53:40.167893922 +0000 2026-02-12T03:54:22.843 DEBUG:teuthology.orchestra.run.trial103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-12T03:54:22.905 INFO:teuthology.orchestra.run.trial103.stderr:1+0 records in 2026-02-12T03:54:22.905 INFO:teuthology.orchestra.run.trial103.stderr:1+0 records out 2026-02-12T03:54:22.905 INFO:teuthology.orchestra.run.trial103.stderr:512 bytes copied, 0.000174853 s, 2.9 MB/s 2026-02-12T03:54:22.906 DEBUG:teuthology.orchestra.run.trial103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-12T03:54:22.965 DEBUG:teuthology.orchestra.run.trial103:> stat /dev/vg_nvme/lv_3 2026-02-12T03:54:23.020 INFO:teuthology.orchestra.run.trial103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-12T03:54:23.021 INFO:teuthology.orchestra.run.trial103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T03:54:23.021 INFO:teuthology.orchestra.run.trial103.stdout:Device: 6h/6d Inode: 1299 Links: 1 2026-02-12T03:54:23.021 INFO:teuthology.orchestra.run.trial103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T03:54:23.021 INFO:teuthology.orchestra.run.trial103.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T03:54:23.021 INFO:teuthology.orchestra.run.trial103.stdout:Access: 2026-02-12 03:54:20.575199991 +0000 2026-02-12T03:54:23.021 INFO:teuthology.orchestra.run.trial103.stdout:Modify: 2026-02-12 03:53:40.167893922 +0000 2026-02-12T03:54:23.021 INFO:teuthology.orchestra.run.trial103.stdout:Change: 2026-02-12 03:53:40.167893922 +0000 2026-02-12T03:54:23.021 INFO:teuthology.orchestra.run.trial103.stdout: Birth: 2026-02-12 03:53:40.167893922 +0000 2026-02-12T03:54:23.022 DEBUG:teuthology.orchestra.run.trial103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-12T03:54:23.084 INFO:teuthology.orchestra.run.trial103.stderr:1+0 records in 2026-02-12T03:54:23.084 INFO:teuthology.orchestra.run.trial103.stderr:1+0 records out 2026-02-12T03:54:23.085 INFO:teuthology.orchestra.run.trial103.stderr:512 bytes copied, 0.000145449 s, 3.5 MB/s 2026-02-12T03:54:23.086 DEBUG:teuthology.orchestra.run.trial103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-12T03:54:23.144 DEBUG:teuthology.orchestra.run.trial103:> stat /dev/vg_nvme/lv_4 2026-02-12T03:54:23.201 INFO:teuthology.orchestra.run.trial103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-12T03:54:23.202 INFO:teuthology.orchestra.run.trial103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T03:54:23.202 INFO:teuthology.orchestra.run.trial103.stdout:Device: 6h/6d Inode: 1289 Links: 1 2026-02-12T03:54:23.202 INFO:teuthology.orchestra.run.trial103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T03:54:23.202 INFO:teuthology.orchestra.run.trial103.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T03:54:23.202 INFO:teuthology.orchestra.run.trial103.stdout:Access: 2026-02-12 03:54:20.576199956 +0000 2026-02-12T03:54:23.202 INFO:teuthology.orchestra.run.trial103.stdout:Modify: 2026-02-12 03:53:40.166893939 +0000 2026-02-12T03:54:23.202 INFO:teuthology.orchestra.run.trial103.stdout:Change: 2026-02-12 03:53:40.166893939 +0000 2026-02-12T03:54:23.203 INFO:teuthology.orchestra.run.trial103.stdout: Birth: 2026-02-12 03:53:40.166893939 +0000 2026-02-12T03:54:23.203 DEBUG:teuthology.orchestra.run.trial103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-12T03:54:23.264 INFO:teuthology.orchestra.run.trial103.stderr:1+0 records in 2026-02-12T03:54:23.264 INFO:teuthology.orchestra.run.trial103.stderr:1+0 records out 2026-02-12T03:54:23.264 INFO:teuthology.orchestra.run.trial103.stderr:512 bytes copied, 0.000133206 s, 3.8 MB/s 2026-02-12T03:54:23.265 DEBUG:teuthology.orchestra.run.trial103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-12T03:54:23.322 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-12T03:54:23.323 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial103.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2026-02-12T03:54:23.323 INFO:tasks.ceph:Generating config... 2026-02-12T03:54:23.325 INFO:tasks.ceph:[osd] osd max pg log entries = 5 2026-02-12T03:54:23.325 INFO:tasks.ceph:[osd] osd min pg log entries = 5 2026-02-12T03:54:23.326 INFO:tasks.ceph:[osd] bdev async discard = True 2026-02-12T03:54:23.326 INFO:tasks.ceph:[osd] bdev enable discard = True 2026-02-12T03:54:23.326 INFO:tasks.ceph:[osd] bluestore allocator = btree 2026-02-12T03:54:23.326 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-12T03:54:23.326 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-12T03:54:23.326 INFO:tasks.ceph:[osd] bluestore onode segment size = 1024K 2026-02-12T03:54:23.326 INFO:tasks.ceph:[osd] bluestore write v2 = True 2026-02-12T03:54:23.326 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-12T03:54:23.326 INFO:tasks.ceph:[osd] debug bluefs = 20 2026-02-12T03:54:23.326 INFO:tasks.ceph:[osd] debug bluestore = 20 2026-02-12T03:54:23.326 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-12T03:54:23.326 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-12T03:54:23.327 INFO:tasks.ceph:[osd] debug rocksdb = 10 2026-02-12T03:54:23.327 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-12T03:54:23.327 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-12T03:54:23.327 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-12T03:54:23.327 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-12T03:54:23.327 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-12T03:54:23.327 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-12T03:54:23.327 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-12T03:54:23.327 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-12T03:54:23.327 INFO:tasks.ceph:[osd] osd mclock skip benchmark = True 2026-02-12T03:54:23.327 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2026-02-12T03:54:23.327 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-12T03:54:23.328 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-12T03:54:23.328 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-12T03:54:23.328 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-02-12T03:54:23.328 INFO:tasks.ceph:[global] mon client hunt interval max multiple = 2 2026-02-12T03:54:23.328 INFO:tasks.ceph:[global] mon election default strategy = 1 2026-02-12T03:54:23.328 INFO:tasks.ceph:[global] mon mgr beacon grace = 90 2026-02-12T03:54:23.328 INFO:tasks.ceph:[global] ms bind msgr1 = False 2026-02-12T03:54:23.328 INFO:tasks.ceph:[global] ms bind msgr2 = True 2026-02-12T03:54:23.328 INFO:tasks.ceph:[global] ms inject socket failures = 1000 2026-02-12T03:54:23.328 INFO:tasks.ceph:[global] ms type = async 2026-02-12T03:54:23.328 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-12T03:54:23.328 INFO:tasks.ceph:[mgr] debug monc = 10 2026-02-12T03:54:23.329 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-12T03:54:23.329 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-12T03:54:23.329 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-12T03:54:23.329 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-12T03:54:23.329 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-12T03:54:23.329 INFO:tasks.ceph:Setting up mon.a... 2026-02-12T03:54:23.329 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-12T03:54:23.397 INFO:teuthology.orchestra.run.trial103.stdout:creating /etc/ceph/ceph.keyring 2026-02-12T03:54:23.399 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-12T03:54:23.476 DEBUG:teuthology.orchestra.run.trial103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-12T03:54:23.542 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.103'), ('mon.b', '[v2:10.20.193.103:3301,v1:10.20.193.103:6790]'), ('mon.c', '[v2:10.20.193.103:3302,v1:10.20.193.103:6791]')] 2026-02-12T03:54:23.543 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.103,[v2:10.20.193.103:3301,v1:10.20.193.103:6790],[v2:10.20.193.103:3302,v1:10.20.193.103:6791]', 'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 1, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'btree', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore onode segment size': '1024K', 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2026-02-12T03:54:23.544 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:23.544 DEBUG:teuthology.orchestra.run.trial103:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-12T03:54:23.599 DEBUG:teuthology.orchestra.run.trial103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.103 --addv b '[v2:10.20.193.103:3301,v1:10.20.193.103:6790]' --addv c '[v2:10.20.193.103:3302,v1:10.20.193.103:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2026-02-12T03:54:23.673 INFO:teuthology.orchestra.run.trial103.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-12T03:54:23.673 INFO:teuthology.orchestra.run.trial103.stdout:monmaptool: generated fsid e9633232-3262-4ad1-a60a-815554f29d9f 2026-02-12T03:54:23.673 INFO:teuthology.orchestra.run.trial103.stdout:setting min_mon_release = tentacle 2026-02-12T03:54:23.673 INFO:teuthology.orchestra.run.trial103.stdout:epoch 0 2026-02-12T03:54:23.673 INFO:teuthology.orchestra.run.trial103.stdout:fsid e9633232-3262-4ad1-a60a-815554f29d9f 2026-02-12T03:54:23.673 INFO:teuthology.orchestra.run.trial103.stdout:last_changed 2026-02-12T03:54:23.672788+0000 2026-02-12T03:54:23.673 INFO:teuthology.orchestra.run.trial103.stdout:created 2026-02-12T03:54:23.672788+0000 2026-02-12T03:54:23.674 INFO:teuthology.orchestra.run.trial103.stdout:min_mon_release 20 (tentacle) 2026-02-12T03:54:23.674 INFO:teuthology.orchestra.run.trial103.stdout:election_strategy: 1 2026-02-12T03:54:23.674 INFO:teuthology.orchestra.run.trial103.stdout:0: [v2:10.20.193.103:3300/0,v1:10.20.193.103:6789/0] mon.a 2026-02-12T03:54:23.674 INFO:teuthology.orchestra.run.trial103.stdout:1: [v2:10.20.193.103:3301/0,v1:10.20.193.103:6790/0] mon.b 2026-02-12T03:54:23.674 INFO:teuthology.orchestra.run.trial103.stdout:2: [v2:10.20.193.103:3302/0,v1:10.20.193.103:6791/0] mon.c 2026-02-12T03:54:23.674 INFO:teuthology.orchestra.run.trial103.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2026-02-12T03:54:23.675 DEBUG:teuthology.orchestra.run.trial103:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-12T03:54:23.731 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e9633232-3262-4ad1-a60a-815554f29d9f... 2026-02-12T03:54:23.732 DEBUG:teuthology.orchestra.run.trial103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-12T03:54:23.809 INFO:teuthology.orchestra.run.trial103.stdout:[global] 2026-02-12T03:54:23.810 INFO:teuthology.orchestra.run.trial103.stdout: chdir = "" 2026-02-12T03:54:23.810 INFO:teuthology.orchestra.run.trial103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-12T03:54:23.810 INFO:teuthology.orchestra.run.trial103.stdout: auth supported = cephx 2026-02-12T03:54:23.810 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.810 INFO:teuthology.orchestra.run.trial103.stdout: filestore xattr use omap = true 2026-02-12T03:54:23.810 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.810 INFO:teuthology.orchestra.run.trial103.stdout: mon clock drift allowed = 1.000 2026-02-12T03:54:23.810 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.810 INFO:teuthology.orchestra.run.trial103.stdout: osd crush chooseleaf type = 0 2026-02-12T03:54:23.811 INFO:teuthology.orchestra.run.trial103.stdout: auth debug = true 2026-02-12T03:54:23.811 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.811 INFO:teuthology.orchestra.run.trial103.stdout: ms die on old message = true 2026-02-12T03:54:23.811 INFO:teuthology.orchestra.run.trial103.stdout: ms die on bug = true 2026-02-12T03:54:23.811 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.811 INFO:teuthology.orchestra.run.trial103.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-12T03:54:23.811 INFO:teuthology.orchestra.run.trial103.stdout: mon pg warn max object skew = 0 2026-02-12T03:54:23.811 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.811 INFO:teuthology.orchestra.run.trial103.stdout: # disable pg_autoscaler by default for new pools 2026-02-12T03:54:23.811 INFO:teuthology.orchestra.run.trial103.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-12T03:54:23.811 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.811 INFO:teuthology.orchestra.run.trial103.stdout: osd pool default size = 2 2026-02-12T03:54:23.812 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.812 INFO:teuthology.orchestra.run.trial103.stdout: mon osd allow primary affinity = true 2026-02-12T03:54:23.812 INFO:teuthology.orchestra.run.trial103.stdout: mon osd allow pg remap = true 2026-02-12T03:54:23.812 INFO:teuthology.orchestra.run.trial103.stdout: mon warn on legacy crush tunables = false 2026-02-12T03:54:23.812 INFO:teuthology.orchestra.run.trial103.stdout: mon warn on crush straw calc version zero = false 2026-02-12T03:54:23.812 INFO:teuthology.orchestra.run.trial103.stdout: mon warn on no sortbitwise = false 2026-02-12T03:54:23.812 INFO:teuthology.orchestra.run.trial103.stdout: mon warn on osd down out interval zero = false 2026-02-12T03:54:23.812 INFO:teuthology.orchestra.run.trial103.stdout: mon warn on too few osds = false 2026-02-12T03:54:23.812 INFO:teuthology.orchestra.run.trial103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-12T03:54:23.812 INFO:teuthology.orchestra.run.trial103.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-12T03:54:23.812 INFO:teuthology.orchestra.run.trial103.stdout: mon_allow_pool_size_one = true 2026-02-12T03:54:23.812 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.813 INFO:teuthology.orchestra.run.trial103.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-12T03:54:23.813 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.813 INFO:teuthology.orchestra.run.trial103.stdout: osd default data pool replay window = 5 2026-02-12T03:54:23.813 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.813 INFO:teuthology.orchestra.run.trial103.stdout: mon allow pool delete = true 2026-02-12T03:54:23.813 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.813 INFO:teuthology.orchestra.run.trial103.stdout: mon cluster log file level = debug 2026-02-12T03:54:23.813 INFO:teuthology.orchestra.run.trial103.stdout: debug asserts on shutdown = true 2026-02-12T03:54:23.813 INFO:teuthology.orchestra.run.trial103.stdout: mon health detail to clog = false 2026-02-12T03:54:23.813 INFO:teuthology.orchestra.run.trial103.stdout: mon host = "10.20.193.103,[v2:10.20.193.103:3301,v1:10.20.193.103:6790],[v2:10.20.193.103:3302,v1:10.20.193.103:6791]" 2026-02-12T03:54:23.813 INFO:teuthology.orchestra.run.trial103.stdout: mon client directed command retry = 5 2026-02-12T03:54:23.813 INFO:teuthology.orchestra.run.trial103.stdout: mon client hunt interval max multiple = 2 2026-02-12T03:54:23.814 INFO:teuthology.orchestra.run.trial103.stdout: mon election default strategy = 1 2026-02-12T03:54:23.814 INFO:teuthology.orchestra.run.trial103.stdout: mon mgr beacon grace = 90 2026-02-12T03:54:23.814 INFO:teuthology.orchestra.run.trial103.stdout: ms bind msgr1 = False 2026-02-12T03:54:23.814 INFO:teuthology.orchestra.run.trial103.stdout: ms bind msgr2 = True 2026-02-12T03:54:23.814 INFO:teuthology.orchestra.run.trial103.stdout: ms inject socket failures = 1000 2026-02-12T03:54:23.814 INFO:teuthology.orchestra.run.trial103.stdout: ms type = async 2026-02-12T03:54:23.814 INFO:teuthology.orchestra.run.trial103.stdout: fsid = e9633232-3262-4ad1-a60a-815554f29d9f 2026-02-12T03:54:23.814 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.814 INFO:teuthology.orchestra.run.trial103.stdout:[osd] 2026-02-12T03:54:23.814 INFO:teuthology.orchestra.run.trial103.stdout: osd journal size = 100 2026-02-12T03:54:23.814 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.814 INFO:teuthology.orchestra.run.trial103.stdout: osd scrub load threshold = 5.0 2026-02-12T03:54:23.814 INFO:teuthology.orchestra.run.trial103.stdout: osd scrub max interval = 600 2026-02-12T03:54:23.815 INFO:teuthology.orchestra.run.trial103.stdout: osd mclock profile = high_recovery_ops 2026-02-12T03:54:23.815 INFO:teuthology.orchestra.run.trial103.stdout: osd mclock skip benchmark = True 2026-02-12T03:54:23.815 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.815 INFO:teuthology.orchestra.run.trial103.stdout: osd recover clone overlap = true 2026-02-12T03:54:23.815 INFO:teuthology.orchestra.run.trial103.stdout: osd recovery max chunk = 1048576 2026-02-12T03:54:23.815 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.815 INFO:teuthology.orchestra.run.trial103.stdout: osd debug shutdown = true 2026-02-12T03:54:23.815 INFO:teuthology.orchestra.run.trial103.stdout: osd debug op order = true 2026-02-12T03:54:23.815 INFO:teuthology.orchestra.run.trial103.stdout: osd debug verify stray on activate = true 2026-02-12T03:54:23.815 INFO:teuthology.orchestra.run.trial103.stdout: osd debug trim objects = true 2026-02-12T03:54:23.815 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.815 INFO:teuthology.orchestra.run.trial103.stdout: osd open classes on start = true 2026-02-12T03:54:23.815 INFO:teuthology.orchestra.run.trial103.stdout: osd debug pg log writeout = true 2026-02-12T03:54:23.816 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.816 INFO:teuthology.orchestra.run.trial103.stdout: osd deep scrub update digest min age = 30 2026-02-12T03:54:23.816 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.816 INFO:teuthology.orchestra.run.trial103.stdout: osd map max advance = 10 2026-02-12T03:54:23.816 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.816 INFO:teuthology.orchestra.run.trial103.stdout: journal zero on create = true 2026-02-12T03:54:23.816 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.816 INFO:teuthology.orchestra.run.trial103.stdout: filestore ondisk finisher threads = 3 2026-02-12T03:54:23.816 INFO:teuthology.orchestra.run.trial103.stdout: filestore apply finisher threads = 3 2026-02-12T03:54:23.816 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.816 INFO:teuthology.orchestra.run.trial103.stdout: bdev debug aio = true 2026-02-12T03:54:23.816 INFO:teuthology.orchestra.run.trial103.stdout: osd debug misdirected ops = true 2026-02-12T03:54:23.817 INFO:teuthology.orchestra.run.trial103.stdout: osd max pg log entries = 5 2026-02-12T03:54:23.817 INFO:teuthology.orchestra.run.trial103.stdout: osd min pg log entries = 5 2026-02-12T03:54:23.817 INFO:teuthology.orchestra.run.trial103.stdout: bdev async discard = True 2026-02-12T03:54:23.817 INFO:teuthology.orchestra.run.trial103.stdout: bdev enable discard = True 2026-02-12T03:54:23.817 INFO:teuthology.orchestra.run.trial103.stdout: bluestore allocator = btree 2026-02-12T03:54:23.817 INFO:teuthology.orchestra.run.trial103.stdout: bluestore block size = 96636764160 2026-02-12T03:54:23.817 INFO:teuthology.orchestra.run.trial103.stdout: bluestore fsck on mount = True 2026-02-12T03:54:23.817 INFO:teuthology.orchestra.run.trial103.stdout: bluestore onode segment size = 1024K 2026-02-12T03:54:23.817 INFO:teuthology.orchestra.run.trial103.stdout: bluestore write v2 = True 2026-02-12T03:54:23.817 INFO:teuthology.orchestra.run.trial103.stdout: bluestore zero block detection = True 2026-02-12T03:54:23.817 INFO:teuthology.orchestra.run.trial103.stdout: debug bluefs = 20 2026-02-12T03:54:23.817 INFO:teuthology.orchestra.run.trial103.stdout: debug bluestore = 20 2026-02-12T03:54:23.817 INFO:teuthology.orchestra.run.trial103.stdout: debug ms = 1 2026-02-12T03:54:23.817 INFO:teuthology.orchestra.run.trial103.stdout: debug osd = 20 2026-02-12T03:54:23.818 INFO:teuthology.orchestra.run.trial103.stdout: debug rocksdb = 10 2026-02-12T03:54:23.818 INFO:teuthology.orchestra.run.trial103.stdout: mon osd backfillfull_ratio = 0.85 2026-02-12T03:54:23.818 INFO:teuthology.orchestra.run.trial103.stdout: mon osd full ratio = 0.9 2026-02-12T03:54:23.818 INFO:teuthology.orchestra.run.trial103.stdout: mon osd nearfull ratio = 0.8 2026-02-12T03:54:23.818 INFO:teuthology.orchestra.run.trial103.stdout: osd debug verify cached snaps = True 2026-02-12T03:54:23.818 INFO:teuthology.orchestra.run.trial103.stdout: osd debug verify missing on start = True 2026-02-12T03:54:23.818 INFO:teuthology.orchestra.run.trial103.stdout: osd failsafe full ratio = 0.95 2026-02-12T03:54:23.818 INFO:teuthology.orchestra.run.trial103.stdout: osd mclock override recovery settings = True 2026-02-12T03:54:23.818 INFO:teuthology.orchestra.run.trial103.stdout: osd memory target = 1610612736 2026-02-12T03:54:23.818 INFO:teuthology.orchestra.run.trial103.stdout: osd objectstore = bluestore 2026-02-12T03:54:23.818 INFO:teuthology.orchestra.run.trial103.stdout: osd op queue = debug_random 2026-02-12T03:54:23.818 INFO:teuthology.orchestra.run.trial103.stdout: osd op queue cut off = debug_random 2026-02-12T03:54:23.818 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.819 INFO:teuthology.orchestra.run.trial103.stdout:[mgr] 2026-02-12T03:54:23.819 INFO:teuthology.orchestra.run.trial103.stdout: debug ms = 1 2026-02-12T03:54:23.819 INFO:teuthology.orchestra.run.trial103.stdout: debug mgr = 20 2026-02-12T03:54:23.819 INFO:teuthology.orchestra.run.trial103.stdout: debug mon = 20 2026-02-12T03:54:23.819 INFO:teuthology.orchestra.run.trial103.stdout: debug auth = 20 2026-02-12T03:54:23.819 INFO:teuthology.orchestra.run.trial103.stdout: mon reweight min pgs per osd = 4 2026-02-12T03:54:23.819 INFO:teuthology.orchestra.run.trial103.stdout: mon reweight min bytes per osd = 10 2026-02-12T03:54:23.819 INFO:teuthology.orchestra.run.trial103.stdout: mgr/telemetry/nag = false 2026-02-12T03:54:23.819 INFO:teuthology.orchestra.run.trial103.stdout: debug monc = 10 2026-02-12T03:54:23.819 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.819 INFO:teuthology.orchestra.run.trial103.stdout:[mon] 2026-02-12T03:54:23.819 INFO:teuthology.orchestra.run.trial103.stdout: debug ms = 1 2026-02-12T03:54:23.819 INFO:teuthology.orchestra.run.trial103.stdout: debug mon = 20 2026-02-12T03:54:23.820 INFO:teuthology.orchestra.run.trial103.stdout: debug paxos = 20 2026-02-12T03:54:23.820 INFO:teuthology.orchestra.run.trial103.stdout: debug auth = 20 2026-02-12T03:54:23.820 INFO:teuthology.orchestra.run.trial103.stdout: mon data avail warn = 5 2026-02-12T03:54:23.820 INFO:teuthology.orchestra.run.trial103.stdout: mon mgr mkfs grace = 240 2026-02-12T03:54:23.820 INFO:teuthology.orchestra.run.trial103.stdout: mon reweight min pgs per osd = 4 2026-02-12T03:54:23.820 INFO:teuthology.orchestra.run.trial103.stdout: mon osd reporter subtree level = osd 2026-02-12T03:54:23.820 INFO:teuthology.orchestra.run.trial103.stdout: mon osd prime pg temp = true 2026-02-12T03:54:23.820 INFO:teuthology.orchestra.run.trial103.stdout: mon reweight min bytes per osd = 10 2026-02-12T03:54:23.820 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.820 INFO:teuthology.orchestra.run.trial103.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-12T03:54:23.820 INFO:teuthology.orchestra.run.trial103.stdout: auth mon ticket ttl = 660 # 11m 2026-02-12T03:54:23.820 INFO:teuthology.orchestra.run.trial103.stdout: auth service ticket ttl = 240 # 4m 2026-02-12T03:54:23.820 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.821 INFO:teuthology.orchestra.run.trial103.stdout: # don't complain about insecure global_id in the test suite 2026-02-12T03:54:23.821 INFO:teuthology.orchestra.run.trial103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-12T03:54:23.821 INFO:teuthology.orchestra.run.trial103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-12T03:54:23.821 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.821 INFO:teuthology.orchestra.run.trial103.stdout: # 1m isn't quite enough 2026-02-12T03:54:23.821 INFO:teuthology.orchestra.run.trial103.stdout: mon_down_mkfs_grace = 2m 2026-02-12T03:54:23.821 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.821 INFO:teuthology.orchestra.run.trial103.stdout: mon_warn_on_filestore_osds = false 2026-02-12T03:54:23.821 INFO:teuthology.orchestra.run.trial103.stdout: mon scrub interval = 300 2026-02-12T03:54:23.821 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:23.821 INFO:teuthology.orchestra.run.trial103.stdout:[client] 2026-02-12T03:54:23.821 INFO:teuthology.orchestra.run.trial103.stdout: rgw cache enabled = true 2026-02-12T03:54:23.822 INFO:teuthology.orchestra.run.trial103.stdout: rgw enable ops log = true 2026-02-12T03:54:23.822 INFO:teuthology.orchestra.run.trial103.stdout: rgw enable usage log = true 2026-02-12T03:54:23.822 INFO:teuthology.orchestra.run.trial103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-12T03:54:23.822 INFO:teuthology.orchestra.run.trial103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-12T03:54:23.822 INFO:teuthology.orchestra.run.trial103.stdout:[mon.a] 2026-02-12T03:54:23.822 INFO:teuthology.orchestra.run.trial103.stdout:[mon.b] 2026-02-12T03:54:23.822 INFO:teuthology.orchestra.run.trial103.stdout:[mon.c] 2026-02-12T03:54:23.823 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-12T03:54:23.824 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-12T03:54:23.905 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-12T03:54:23.906 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:23.906 DEBUG:teuthology.orchestra.run.trial103:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-12T03:54:23.962 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:23.962 DEBUG:teuthology.orchestra.run.trial103:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-12T03:54:24.018 INFO:tasks.ceph:Sending monmap to node ubuntu@trial103.front.sepia.ceph.com 2026-02-12T03:54:24.018 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:24.018 DEBUG:teuthology.orchestra.run.trial103:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-12T03:54:24.019 DEBUG:teuthology.orchestra.run.trial103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-12T03:54:24.089 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:24.089 DEBUG:teuthology.orchestra.run.trial103:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-12T03:54:24.147 INFO:tasks.ceph:Setting up mon nodes... 2026-02-12T03:54:24.148 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-12T03:54:24.148 DEBUG:teuthology.orchestra.run.trial103:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-12T03:54:24.230 INFO:teuthology.orchestra.run.trial103.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-12T03:54:24.233 INFO:tasks.ceph:Setting up mds nodes... 2026-02-12T03:54:24.233 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-12T03:54:24.233 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-12T03:54:24.233 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial103.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2026-02-12T03:54:24.233 DEBUG:teuthology.orchestra.run.trial103:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-12T03:54:24.298 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-12T03:54:24.298 INFO:tasks.ceph:role: osd.0 2026-02-12T03:54:24.298 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial103.front.sepia.ceph.com 2026-02-12T03:54:24.298 DEBUG:teuthology.orchestra.run.trial103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-12T03:54:24.362 INFO:teuthology.orchestra.run.trial103.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T03:54:24.362 INFO:teuthology.orchestra.run.trial103.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T03:54:24.362 INFO:teuthology.orchestra.run.trial103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T03:54:24.363 INFO:teuthology.orchestra.run.trial103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T03:54:24.363 INFO:teuthology.orchestra.run.trial103.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T03:54:24.363 INFO:teuthology.orchestra.run.trial103.stdout: = sunit=0 swidth=0 blks 2026-02-12T03:54:24.363 INFO:teuthology.orchestra.run.trial103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T03:54:24.363 INFO:teuthology.orchestra.run.trial103.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T03:54:24.363 INFO:teuthology.orchestra.run.trial103.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T03:54:24.363 INFO:teuthology.orchestra.run.trial103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T03:54:24.943 INFO:teuthology.orchestra.run.trial103.stdout:Discarding blocks...Done. 2026-02-12T03:54:24.946 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial103.front.sepia.ceph.com -o noatime 2026-02-12T03:54:24.946 DEBUG:teuthology.orchestra.run.trial103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-12T03:54:25.013 DEBUG:teuthology.orchestra.run.trial103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-12T03:54:25.080 DEBUG:teuthology.orchestra.run.trial103:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-12T03:54:25.143 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-12T03:54:25.144 INFO:tasks.ceph:role: osd.1 2026-02-12T03:54:25.144 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial103.front.sepia.ceph.com 2026-02-12T03:54:25.144 DEBUG:teuthology.orchestra.run.trial103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-12T03:54:25.209 INFO:teuthology.orchestra.run.trial103.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T03:54:25.209 INFO:teuthology.orchestra.run.trial103.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T03:54:25.209 INFO:teuthology.orchestra.run.trial103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T03:54:25.209 INFO:teuthology.orchestra.run.trial103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T03:54:25.209 INFO:teuthology.orchestra.run.trial103.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T03:54:25.209 INFO:teuthology.orchestra.run.trial103.stdout: = sunit=0 swidth=0 blks 2026-02-12T03:54:25.210 INFO:teuthology.orchestra.run.trial103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T03:54:25.210 INFO:teuthology.orchestra.run.trial103.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T03:54:25.210 INFO:teuthology.orchestra.run.trial103.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T03:54:25.210 INFO:teuthology.orchestra.run.trial103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T03:54:25.768 INFO:teuthology.orchestra.run.trial103.stdout:Discarding blocks...Done. 2026-02-12T03:54:25.770 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial103.front.sepia.ceph.com -o noatime 2026-02-12T03:54:25.770 DEBUG:teuthology.orchestra.run.trial103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-12T03:54:25.837 DEBUG:teuthology.orchestra.run.trial103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-12T03:54:25.904 DEBUG:teuthology.orchestra.run.trial103:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-12T03:54:25.967 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-12T03:54:25.967 INFO:tasks.ceph:role: osd.2 2026-02-12T03:54:25.967 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial103.front.sepia.ceph.com 2026-02-12T03:54:25.967 DEBUG:teuthology.orchestra.run.trial103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-12T03:54:26.033 INFO:teuthology.orchestra.run.trial103.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T03:54:26.033 INFO:teuthology.orchestra.run.trial103.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T03:54:26.033 INFO:teuthology.orchestra.run.trial103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T03:54:26.034 INFO:teuthology.orchestra.run.trial103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T03:54:26.034 INFO:teuthology.orchestra.run.trial103.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T03:54:26.034 INFO:teuthology.orchestra.run.trial103.stdout: = sunit=0 swidth=0 blks 2026-02-12T03:54:26.034 INFO:teuthology.orchestra.run.trial103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T03:54:26.034 INFO:teuthology.orchestra.run.trial103.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T03:54:26.034 INFO:teuthology.orchestra.run.trial103.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T03:54:26.034 INFO:teuthology.orchestra.run.trial103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T03:54:26.598 INFO:teuthology.orchestra.run.trial103.stdout:Discarding blocks...Done. 2026-02-12T03:54:26.600 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial103.front.sepia.ceph.com -o noatime 2026-02-12T03:54:26.600 DEBUG:teuthology.orchestra.run.trial103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-12T03:54:26.669 DEBUG:teuthology.orchestra.run.trial103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-12T03:54:26.735 DEBUG:teuthology.orchestra.run.trial103:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-02-12T03:54:26.798 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-12T03:54:26.799 INFO:tasks.ceph:role: osd.3 2026-02-12T03:54:26.799 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@trial103.front.sepia.ceph.com 2026-02-12T03:54:26.799 DEBUG:teuthology.orchestra.run.trial103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2026-02-12T03:54:26.863 INFO:teuthology.orchestra.run.trial103.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T03:54:26.863 INFO:teuthology.orchestra.run.trial103.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T03:54:26.863 INFO:teuthology.orchestra.run.trial103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T03:54:26.863 INFO:teuthology.orchestra.run.trial103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T03:54:26.863 INFO:teuthology.orchestra.run.trial103.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T03:54:26.863 INFO:teuthology.orchestra.run.trial103.stdout: = sunit=0 swidth=0 blks 2026-02-12T03:54:26.864 INFO:teuthology.orchestra.run.trial103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T03:54:26.864 INFO:teuthology.orchestra.run.trial103.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T03:54:26.864 INFO:teuthology.orchestra.run.trial103.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T03:54:26.864 INFO:teuthology.orchestra.run.trial103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T03:54:27.418 INFO:teuthology.orchestra.run.trial103.stdout:Discarding blocks...Done. 2026-02-12T03:54:27.420 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@trial103.front.sepia.ceph.com -o noatime 2026-02-12T03:54:27.420 DEBUG:teuthology.orchestra.run.trial103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2026-02-12T03:54:27.494 DEBUG:teuthology.orchestra.run.trial103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-02-12T03:54:27.560 DEBUG:teuthology.orchestra.run.trial103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T03:54:27.639 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:27.637+0000 7f6928581900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-02-12T03:54:27.639 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:27.637+0000 7f6928581900 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-02-12T03:54:27.639 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:27.637+0000 7f6928581900 -1 bdev(0x55c8e997b800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-02-12T03:54:27.640 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:27.637+0000 7f6928581900 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-02-12T03:54:28.044 DEBUG:teuthology.orchestra.run.trial103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-12T03:54:28.110 DEBUG:teuthology.orchestra.run.trial103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T03:54:28.188 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:28.186+0000 7f540060d900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-02-12T03:54:28.188 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:28.186+0000 7f540060d900 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-02-12T03:54:28.188 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:28.186+0000 7f540060d900 -1 bdev(0x561c6a999800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-02-12T03:54:28.188 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:28.186+0000 7f540060d900 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-02-12T03:54:28.590 DEBUG:teuthology.orchestra.run.trial103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-12T03:54:28.655 DEBUG:teuthology.orchestra.run.trial103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T03:54:28.735 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:28.733+0000 7f42b4d5f900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-02-12T03:54:28.735 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:28.733+0000 7f42b4d5f900 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-02-12T03:54:28.736 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:28.733+0000 7f42b4d5f900 -1 bdev(0x55963ba9d800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-02-12T03:54:28.736 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:28.733+0000 7f42b4d5f900 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-02-12T03:54:29.139 DEBUG:teuthology.orchestra.run.trial103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-12T03:54:29.204 DEBUG:teuthology.orchestra.run.trial103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T03:54:29.283 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:29.281+0000 7fba1d193900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2026-02-12T03:54:29.284 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:29.281+0000 7fba1d193900 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2026-02-12T03:54:29.284 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:29.281+0000 7fba1d193900 -1 bdev(0x55f2d4981800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2026-02-12T03:54:29.284 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:29.282+0000 7fba1d193900 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2026-02-12T03:54:29.659 DEBUG:teuthology.orchestra.run.trial103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-12T03:54:29.724 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-12T03:54:29.724 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:29.724 DEBUG:teuthology.orchestra.run.trial103:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-12T03:54:29.788 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:29.789 DEBUG:teuthology.orchestra.run.trial103:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-12T03:54:29.853 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:29.853 DEBUG:teuthology.orchestra.run.trial103:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-12T03:54:29.915 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:29.915 DEBUG:teuthology.orchestra.run.trial103:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-12T03:54:29.978 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:29.978 DEBUG:teuthology.orchestra.run.trial103:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-02-12T03:54:30.041 INFO:tasks.ceph:Adding keys to all mons... 2026-02-12T03:54:30.041 DEBUG:teuthology.orchestra.run.trial103:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-12T03:54:30.103 INFO:teuthology.orchestra.run.trial103.stdout:[mgr.x] 2026-02-12T03:54:30.103 INFO:teuthology.orchestra.run.trial103.stdout: key = AQDwTo1pVuu8DRAAQklUaW+J7vMoL/LVs+xb9g== 2026-02-12T03:54:30.103 INFO:teuthology.orchestra.run.trial103.stdout:[osd.0] 2026-02-12T03:54:30.103 INFO:teuthology.orchestra.run.trial103.stdout: key = AQDzTo1phjwUJhAAksqeSAT+cwH2iR3Xz1Ai0g== 2026-02-12T03:54:30.103 INFO:teuthology.orchestra.run.trial103.stdout:[osd.1] 2026-02-12T03:54:30.104 INFO:teuthology.orchestra.run.trial103.stdout: key = AQD0To1pfCY0CxAAF8mqA9OCMLZOa3a5tCG7BA== 2026-02-12T03:54:30.104 INFO:teuthology.orchestra.run.trial103.stdout:[osd.2] 2026-02-12T03:54:30.104 INFO:teuthology.orchestra.run.trial103.stdout: key = AQD0To1pMLvQKxAAQkhksuffVkgGShy7nuz74A== 2026-02-12T03:54:30.104 INFO:teuthology.orchestra.run.trial103.stdout:[osd.3] 2026-02-12T03:54:30.104 INFO:teuthology.orchestra.run.trial103.stdout: key = AQD1To1pzaXhEBAAJZVmuqLvXaFes4axoewZdg== 2026-02-12T03:54:30.105 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-12T03:54:30.184 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T03:54:30.264 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T03:54:30.343 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T03:54:30.424 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T03:54:30.501 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-12T03:54:30.502 DEBUG:teuthology.orchestra.run.trial103:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-12T03:54:30.566 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-12T03:54:30.648 DEBUG:teuthology.orchestra.run.trial103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-12T03:54:30.712 DEBUG:teuthology.orchestra.run.trial103:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2026-02-12T03:54:30.775 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-12T03:54:30.859 DEBUG:teuthology.orchestra.run.trial103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2026-02-12T03:54:30.922 DEBUG:teuthology.orchestra.run.trial103:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2026-02-12T03:54:30.986 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-12T03:54:31.069 DEBUG:teuthology.orchestra.run.trial103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2026-02-12T03:54:31.133 DEBUG:teuthology.orchestra.run.trial103:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-12T03:54:31.189 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-12T03:54:31.189 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-12T03:54:31.190 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-12T03:54:31.231 INFO:tasks.ceph.mon.a:Started 2026-02-12T03:54:31.231 INFO:tasks.ceph.mon.b:Restarting daemon 2026-02-12T03:54:31.231 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2026-02-12T03:54:31.234 INFO:tasks.ceph.mon.b:Started 2026-02-12T03:54:31.234 INFO:tasks.ceph.mon.c:Restarting daemon 2026-02-12T03:54:31.234 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2026-02-12T03:54:31.236 INFO:tasks.ceph.mon.c:Started 2026-02-12T03:54:31.236 DEBUG:teuthology.orchestra.run.trial103:> sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force 2026-02-12T03:54:37.348 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-12T03:54:37.349 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-12T03:54:37.349 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-12T03:54:37.351 INFO:tasks.ceph.mgr.x:Started 2026-02-12T03:54:37.352 DEBUG:tasks.ceph:set 0 configs 2026-02-12T03:54:37.352 DEBUG:teuthology.orchestra.run.trial103:> sudo ceph --cluster ceph config dump 2026-02-12T03:54:37.448 INFO:teuthology.orchestra.run.trial103.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-12T03:54:37.448 INFO:teuthology.orchestra.run.trial103.stdout:mgr unknown mgr/devicehealth/enable_monitoring false * 2026-02-12T03:54:37.455 INFO:tasks.ceph:Setting crush tunables to default 2026-02-12T03:54:37.455 DEBUG:teuthology.orchestra.run.trial103:> sudo ceph --cluster ceph osd crush tunables default 2026-02-12T03:54:37.559 INFO:teuthology.orchestra.run.trial103.stderr:adjusted tunables profile to default 2026-02-12T03:54:37.566 INFO:tasks.ceph:check_enable_crimson: False 2026-02-12T03:54:37.566 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-12T03:54:37.566 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:37.566 DEBUG:teuthology.orchestra.run.trial103:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-12T03:54:37.591 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:37.591 DEBUG:teuthology.orchestra.run.trial103:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-12T03:54:37.656 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:37.656 DEBUG:teuthology.orchestra.run.trial103:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-12T03:54:37.719 DEBUG:teuthology.orchestra.run.trial103:> set -ex 2026-02-12T03:54:37.719 DEBUG:teuthology.orchestra.run.trial103:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-02-12T03:54:37.783 DEBUG:teuthology.orchestra.run.trial103:> sudo ceph --cluster ceph osd new 5abdfff6-6d88-4b2f-b2d5-4ef7df15e16f 0 2026-02-12T03:54:37.926 INFO:teuthology.orchestra.run.trial103.stdout:0 2026-02-12T03:54:37.934 DEBUG:teuthology.orchestra.run.trial103:> sudo ceph --cluster ceph osd new 619e1a4c-89fd-4b3f-89b2-59b67eff290c 1 2026-02-12T03:54:38.041 INFO:teuthology.orchestra.run.trial103.stdout:1 2026-02-12T03:54:38.050 DEBUG:teuthology.orchestra.run.trial103:> sudo ceph --cluster ceph osd new d4795e0e-f032-4fa5-a993-f36b7e5b497f 2 2026-02-12T03:54:38.156 INFO:teuthology.orchestra.run.trial103.stdout:2 2026-02-12T03:54:38.165 DEBUG:teuthology.orchestra.run.trial103:> sudo ceph --cluster ceph osd new 4d822481-0e6c-42a6-b420-d6885a3f6cc2 3 2026-02-12T03:54:38.264 INFO:tasks.ceph.mgr.x.trial103.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-12T03:54:38.265 INFO:tasks.ceph.mgr.x.trial103.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-12T03:54:38.265 INFO:tasks.ceph.mgr.x.trial103.stderr: from numpy import show_config as show_numpy_config 2026-02-12T03:54:38.269 INFO:teuthology.orchestra.run.trial103.stdout:3 2026-02-12T03:54:38.278 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-12T03:54:38.278 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-12T03:54:38.280 INFO:tasks.ceph.osd.0:Started 2026-02-12T03:54:38.280 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-12T03:54:38.281 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-12T03:54:38.284 INFO:tasks.ceph.osd.1:Started 2026-02-12T03:54:38.284 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-12T03:54:38.284 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-12T03:54:38.287 INFO:tasks.ceph.osd.2:Started 2026-02-12T03:54:38.287 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-12T03:54:38.288 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2026-02-12T03:54:38.291 INFO:tasks.ceph.osd.3:Started 2026-02-12T03:54:38.291 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T03:54:38.394 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:38.394 INFO:teuthology.orchestra.run.trial103.stdout:{"epoch":6,"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","created":"2026-02-12T03:54:36.502189+0000","modified":"2026-02-12T03:54:38.267568+0000","last_up_change":"0.000000","last_in_change":"2026-02-12T03:54:38.267568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5abdfff6-6d88-4b2f-b2d5-4ef7df15e16f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"619e1a4c-89fd-4b3f-89b2-59b67eff290c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d4795e0e-f032-4fa5-a993-f36b7e5b497f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4d822481-0e6c-42a6-b420-d6885a3f6cc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T03:54:38.400 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-12T03:54:38.401 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-12T03:54:38.591 INFO:tasks.ceph.osd.0.trial103.stderr:2026-02-12T03:54:38.590+0000 7ff7a8990900 -1 Falling back to public interface 2026-02-12T03:54:38.599 INFO:tasks.ceph.osd.3.trial103.stderr:2026-02-12T03:54:38.598+0000 7f481556b900 -1 Falling back to public interface 2026-02-12T03:54:38.607 INFO:tasks.ceph.osd.1.trial103.stderr:2026-02-12T03:54:38.606+0000 7f4c061c8900 -1 Falling back to public interface 2026-02-12T03:54:38.607 INFO:tasks.ceph.osd.2.trial103.stderr:2026-02-12T03:54:38.606+0000 7f7587f44900 -1 Falling back to public interface 2026-02-12T03:54:38.804 DEBUG:teuthology.orchestra.run.trial103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-12T03:54:38.897 INFO:teuthology.misc.health.trial103.stdout: 2026-02-12T03:54:38.897 INFO:teuthology.misc.health.trial103.stdout:{"epoch":6,"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","created":"2026-02-12T03:54:36.502189+0000","modified":"2026-02-12T03:54:38.267568+0000","last_up_change":"0.000000","last_in_change":"2026-02-12T03:54:38.267568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5abdfff6-6d88-4b2f-b2d5-4ef7df15e16f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"619e1a4c-89fd-4b3f-89b2-59b67eff290c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d4795e0e-f032-4fa5-a993-f36b7e5b497f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4d822481-0e6c-42a6-b420-d6885a3f6cc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T03:54:38.905 DEBUG:teuthology.misc:0 of 4 OSDs are up 2026-02-12T03:54:39.208 INFO:tasks.ceph.osd.0.trial103.stderr:2026-02-12T03:54:39.207+0000 7ff7a8990900 -1 osd.0 0 log_to_monitors true 2026-02-12T03:54:39.215 INFO:tasks.ceph.osd.2.trial103.stderr:2026-02-12T03:54:39.214+0000 7f7587f44900 -1 osd.2 0 log_to_monitors true 2026-02-12T03:54:39.221 INFO:tasks.ceph.osd.3.trial103.stderr:2026-02-12T03:54:39.220+0000 7f481556b900 -1 osd.3 0 log_to_monitors true 2026-02-12T03:54:39.226 INFO:tasks.ceph.osd.1.trial103.stderr:2026-02-12T03:54:39.225+0000 7f4c061c8900 -1 osd.1 0 log_to_monitors true 2026-02-12T03:54:40.512 INFO:tasks.ceph.osd.0.trial103.stderr:2026-02-12T03:54:40.511+0000 7ff7a40fb640 -1 osd.0 0 waiting for initial osdmap 2026-02-12T03:54:40.513 INFO:tasks.ceph.osd.3.trial103.stderr:2026-02-12T03:54:40.511+0000 7f4810cd6640 -1 osd.3 0 waiting for initial osdmap 2026-02-12T03:54:40.513 INFO:tasks.ceph.osd.1.trial103.stderr:2026-02-12T03:54:40.511+0000 7f4c009b6640 -1 osd.1 0 waiting for initial osdmap 2026-02-12T03:54:40.513 INFO:tasks.ceph.osd.2.trial103.stderr:2026-02-12T03:54:40.513+0000 7f75836af640 -1 osd.2 0 waiting for initial osdmap 2026-02-12T03:54:40.514 INFO:tasks.ceph.osd.0.trial103.stderr:2026-02-12T03:54:40.513+0000 7ff79ff13640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-12T03:54:40.514 INFO:tasks.ceph.osd.1.trial103.stderr:2026-02-12T03:54:40.513+0000 7f4bf87c6640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-12T03:54:40.515 INFO:tasks.ceph.osd.3.trial103.stderr:2026-02-12T03:54:40.513+0000 7f4808ae6640 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-12T03:54:40.515 INFO:tasks.ceph.osd.2.trial103.stderr:2026-02-12T03:54:40.514+0000 7f757b4bf640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-12T03:54:42.661 INFO:tasks.ceph.mgr.x.trial103.stderr:2026-02-12T03:54:42.659+0000 7feca92d4640 -1 mgr.server handle_report got status from non-daemon mon.c 2026-02-12T03:54:42.661 INFO:tasks.ceph.mgr.x.trial103.stderr:2026-02-12T03:54:42.660+0000 7feca92d4640 -1 mgr.server handle_report got status from non-daemon mon.b 2026-02-12T03:54:45.307 DEBUG:teuthology.orchestra.run.trial103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-12T03:54:45.484 INFO:teuthology.misc.health.trial103.stdout: 2026-02-12T03:54:45.484 INFO:teuthology.misc.health.trial103.stdout:{"epoch":10,"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","created":"2026-02-12T03:54:36.502189+0000","modified":"2026-02-12T03:54:44.663759+0000","last_up_change":"2026-02-12T03:54:41.512748+0000","last_in_change":"2026-02-12T03:54:38.267568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T03:54:43.668614+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5abdfff6-6d88-4b2f-b2d5-4ef7df15e16f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6800","nonce":3386817029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6801","nonce":3386817029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6803","nonce":3386817029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6802","nonce":3386817029}]},"public_addr":"10.20.193.103:6800/3386817029","cluster_addr":"10.20.193.103:6801/3386817029","heartbeat_back_addr":"10.20.193.103:6803/3386817029","heartbeat_front_addr":"10.20.193.103:6802/3386817029","state":["exists","up"]},{"osd":1,"uuid":"619e1a4c-89fd-4b3f-89b2-59b67eff290c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6808","nonce":38137239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6810","nonce":38137239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6814","nonce":38137239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6812","nonce":38137239}]},"public_addr":"10.20.193.103:6808/38137239","cluster_addr":"10.20.193.103:6810/38137239","heartbeat_back_addr":"10.20.193.103:6814/38137239","heartbeat_front_addr":"10.20.193.103:6812/38137239","state":["exists","up"]},{"osd":2,"uuid":"d4795e0e-f032-4fa5-a993-f36b7e5b497f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6809","nonce":2960779923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6811","nonce":2960779923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6815","nonce":2960779923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6813","nonce":2960779923}]},"public_addr":"10.20.193.103:6809/2960779923","cluster_addr":"10.20.193.103:6811/2960779923","heartbeat_back_addr":"10.20.193.103:6815/2960779923","heartbeat_front_addr":"10.20.193.103:6813/2960779923","state":["exists","up"]},{"osd":3,"uuid":"4d822481-0e6c-42a6-b420-d6885a3f6cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6804","nonce":3177587099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6805","nonce":3177587099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6807","nonce":3177587099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6806","nonce":3177587099}]},"public_addr":"10.20.193.103:6804/3177587099","cluster_addr":"10.20.193.103:6805/3177587099","heartbeat_back_addr":"10.20.193.103:6807/3177587099","heartbeat_front_addr":"10.20.193.103:6806/3177587099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T03:54:45.491 DEBUG:teuthology.misc:4 of 4 OSDs are up 2026-02-12T03:54:45.491 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-12T03:54:45.492 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-12T03:54:45.493 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-12T03:54:45.676 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:45.687 INFO:teuthology.orchestra.run.trial103.stdout:[{"version":2,"timestamp":"2026-02-12T03:54:37.338763+0000","name":"","changes":[{"name":"mgr/mgr/devicehealth/enable_monitoring","new_value":"false"}]}] 2026-02-12T03:54:45.687 INFO:tasks.ceph_manager:config epoch is 2 2026-02-12T03:54:45.687 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-12T03:54:45.687 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-12T03:54:45.688 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-12T03:54:45.903 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:45.911 INFO:teuthology.orchestra.run.trial103.stdout:{"epoch":4,"flags":0,"active_gid":4109,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6816","nonce":2117898369}]},"active_addr":"10.20.193.103:6816/2117898369","active_change":"2026-02-12T03:54:41.653181+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self certificates generated and signed by cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.5","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"how frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.103:0","nonce":1151912463}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.103:0","nonce":4215057112}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.103:0","nonce":4048178507}]}]} 2026-02-12T03:54:45.914 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-12T03:54:45.915 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-12T03:54:45.915 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T03:54:46.096 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:46.096 INFO:teuthology.orchestra.run.trial103.stdout:{"epoch":11,"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","created":"2026-02-12T03:54:36.502189+0000","modified":"2026-02-12T03:54:45.666459+0000","last_up_change":"2026-02-12T03:54:41.512748+0000","last_in_change":"2026-02-12T03:54:38.267568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T03:54:43.668614+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5abdfff6-6d88-4b2f-b2d5-4ef7df15e16f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6800","nonce":3386817029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6801","nonce":3386817029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6803","nonce":3386817029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6802","nonce":3386817029}]},"public_addr":"10.20.193.103:6800/3386817029","cluster_addr":"10.20.193.103:6801/3386817029","heartbeat_back_addr":"10.20.193.103:6803/3386817029","heartbeat_front_addr":"10.20.193.103:6802/3386817029","state":["exists","up"]},{"osd":1,"uuid":"619e1a4c-89fd-4b3f-89b2-59b67eff290c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6808","nonce":38137239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6810","nonce":38137239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6814","nonce":38137239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6812","nonce":38137239}]},"public_addr":"10.20.193.103:6808/38137239","cluster_addr":"10.20.193.103:6810/38137239","heartbeat_back_addr":"10.20.193.103:6814/38137239","heartbeat_front_addr":"10.20.193.103:6812/38137239","state":["exists","up"]},{"osd":2,"uuid":"d4795e0e-f032-4fa5-a993-f36b7e5b497f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6809","nonce":2960779923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6811","nonce":2960779923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6815","nonce":2960779923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6813","nonce":2960779923}]},"public_addr":"10.20.193.103:6809/2960779923","cluster_addr":"10.20.193.103:6811/2960779923","heartbeat_back_addr":"10.20.193.103:6815/2960779923","heartbeat_front_addr":"10.20.193.103:6813/2960779923","state":["exists","up"]},{"osd":3,"uuid":"4d822481-0e6c-42a6-b420-d6885a3f6cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6804","nonce":3177587099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6805","nonce":3177587099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6807","nonce":3177587099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6806","nonce":3177587099}]},"public_addr":"10.20.193.103:6804/3177587099","cluster_addr":"10.20.193.103:6805/3177587099","heartbeat_back_addr":"10.20.193.103:6807/3177587099","heartbeat_front_addr":"10.20.193.103:6806/3177587099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T03:54:46.105 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-12T03:54:46.105 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T03:54:46.285 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:46.285 INFO:teuthology.orchestra.run.trial103.stdout:{"epoch":11,"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","created":"2026-02-12T03:54:36.502189+0000","modified":"2026-02-12T03:54:45.666459+0000","last_up_change":"2026-02-12T03:54:41.512748+0000","last_in_change":"2026-02-12T03:54:38.267568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T03:54:43.668614+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5abdfff6-6d88-4b2f-b2d5-4ef7df15e16f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6800","nonce":3386817029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6801","nonce":3386817029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6803","nonce":3386817029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6802","nonce":3386817029}]},"public_addr":"10.20.193.103:6800/3386817029","cluster_addr":"10.20.193.103:6801/3386817029","heartbeat_back_addr":"10.20.193.103:6803/3386817029","heartbeat_front_addr":"10.20.193.103:6802/3386817029","state":["exists","up"]},{"osd":1,"uuid":"619e1a4c-89fd-4b3f-89b2-59b67eff290c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6808","nonce":38137239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6810","nonce":38137239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6814","nonce":38137239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6812","nonce":38137239}]},"public_addr":"10.20.193.103:6808/38137239","cluster_addr":"10.20.193.103:6810/38137239","heartbeat_back_addr":"10.20.193.103:6814/38137239","heartbeat_front_addr":"10.20.193.103:6812/38137239","state":["exists","up"]},{"osd":2,"uuid":"d4795e0e-f032-4fa5-a993-f36b7e5b497f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6809","nonce":2960779923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6811","nonce":2960779923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6815","nonce":2960779923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6813","nonce":2960779923}]},"public_addr":"10.20.193.103:6809/2960779923","cluster_addr":"10.20.193.103:6811/2960779923","heartbeat_back_addr":"10.20.193.103:6815/2960779923","heartbeat_front_addr":"10.20.193.103:6813/2960779923","state":["exists","up"]},{"osd":3,"uuid":"4d822481-0e6c-42a6-b420-d6885a3f6cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6804","nonce":3177587099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6805","nonce":3177587099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6807","nonce":3177587099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6806","nonce":3177587099}]},"public_addr":"10.20.193.103:6804/3177587099","cluster_addr":"10.20.193.103:6805/3177587099","heartbeat_back_addr":"10.20.193.103:6807/3177587099","heartbeat_front_addr":"10.20.193.103:6806/3177587099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T03:54:46.295 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-12T03:54:46.295 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-12T03:54:46.296 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-12T03:54:46.296 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-12T03:54:46.384 INFO:teuthology.orchestra.run.trial103.stdout:38654705666 2026-02-12T03:54:46.385 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-12T03:54:46.386 INFO:teuthology.orchestra.run.trial103.stdout:38654705666 2026-02-12T03:54:46.386 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-12T03:54:46.387 INFO:teuthology.orchestra.run.trial103.stdout:38654705666 2026-02-12T03:54:46.388 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-12T03:54:46.389 INFO:teuthology.orchestra.run.trial103.stdout:38654705666 2026-02-12T03:54:46.389 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-12T03:54:46.607 INFO:teuthology.orchestra.run.trial103.stdout:38654705665 2026-02-12T03:54:46.608 INFO:teuthology.orchestra.run.trial103.stdout:38654705665 2026-02-12T03:54:46.609 INFO:teuthology.orchestra.run.trial103.stdout:38654705665 2026-02-12T03:54:46.610 INFO:teuthology.orchestra.run.trial103.stdout:38654705665 2026-02-12T03:54:46.620 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.3 2026-02-12T03:54:46.621 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.0 2026-02-12T03:54:46.623 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.2 2026-02-12T03:54:46.624 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.1 2026-02-12T03:54:47.621 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-12T03:54:47.622 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-12T03:54:47.624 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-12T03:54:47.625 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-12T03:54:47.807 INFO:teuthology.orchestra.run.trial103.stdout:38654705666 2026-02-12T03:54:47.812 INFO:teuthology.orchestra.run.trial103.stdout:38654705666 2026-02-12T03:54:47.818 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.3 2026-02-12T03:54:47.819 DEBUG:teuthology.parallel:result is None 2026-02-12T03:54:47.823 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.0 2026-02-12T03:54:47.823 DEBUG:teuthology.parallel:result is None 2026-02-12T03:54:47.848 INFO:teuthology.orchestra.run.trial103.stdout:38654705666 2026-02-12T03:54:47.855 INFO:teuthology.orchestra.run.trial103.stdout:38654705666 2026-02-12T03:54:47.859 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.1 2026-02-12T03:54:47.860 DEBUG:teuthology.parallel:result is None 2026-02-12T03:54:47.866 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.2 2026-02-12T03:54:47.867 DEBUG:teuthology.parallel:result is None 2026-02-12T03:54:47.867 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-12T03:54:47.867 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:54:48.086 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:48.086 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:54:48.095 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2026-02-12T03:54:47.657072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":2,"kb":377487360,"kb_used":108872,"kb_used_data":1032,"kb_used_omap":21,"kb_used_meta":107754,"kb_avail":377378488,"statfs":{"total":386547056640,"available":386435571712,"internally_reserved":0,"allocated":1056768,"data_stored":745160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22073,"internal_metadata":110340551},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.983519"},"pg_stats":[{"pgid":"1.0","version":"11'17","reported_seq":33,"reported_epoch":11,"state":"active+clean","last_fresh":"2026-02-12T03:54:46.129819+0000","last_change":"2026-02-12T03:54:46.118818+0000","last_active":"2026-02-12T03:54:46.129819+0000","last_peered":"2026-02-12T03:54:46.129819+0000","last_clean":"2026-02-12T03:54:46.129819+0000","last_became_active":"2026-02-12T03:54:46.118306+0000","last_became_peered":"2026-02-12T03:54:46.118306+0000","last_unstale":"2026-02-12T03:54:46.129819+0000","last_undegraded":"2026-02-12T03:54:46.129819+0000","last_fullsized":"2026-02-12T03:54:46.129819+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T06:40:49.263568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27056,"kb_used_data":96,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":94344784,"statfs":{"total":96636764160,"available":96609058816,"internally_reserved":0,"allocated":98304,"data_stored":22194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5517,"internal_metadata":27585139},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27056,"kb_used_data":96,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":94344784,"statfs":{"total":96636764160,"available":96609058816,"internally_reserved":0,"allocated":98304,"data_stored":22194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5519,"internal_metadata":27585137},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":420,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":430080,"data_stored":350386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5518,"internal_metadata":27585138},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":420,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":430080,"data_stored":350386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5519,"internal_metadata":27585137},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:54:48.096 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:54:48.273 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:48.274 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:54:48.282 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2026-02-12T03:54:47.657072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":2,"kb":377487360,"kb_used":108872,"kb_used_data":1032,"kb_used_omap":21,"kb_used_meta":107754,"kb_avail":377378488,"statfs":{"total":386547056640,"available":386435571712,"internally_reserved":0,"allocated":1056768,"data_stored":745160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22073,"internal_metadata":110340551},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.983519"},"pg_stats":[{"pgid":"1.0","version":"11'17","reported_seq":33,"reported_epoch":11,"state":"active+clean","last_fresh":"2026-02-12T03:54:46.129819+0000","last_change":"2026-02-12T03:54:46.118818+0000","last_active":"2026-02-12T03:54:46.129819+0000","last_peered":"2026-02-12T03:54:46.129819+0000","last_clean":"2026-02-12T03:54:46.129819+0000","last_became_active":"2026-02-12T03:54:46.118306+0000","last_became_peered":"2026-02-12T03:54:46.118306+0000","last_unstale":"2026-02-12T03:54:46.129819+0000","last_undegraded":"2026-02-12T03:54:46.129819+0000","last_fullsized":"2026-02-12T03:54:46.129819+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T06:40:49.263568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27056,"kb_used_data":96,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":94344784,"statfs":{"total":96636764160,"available":96609058816,"internally_reserved":0,"allocated":98304,"data_stored":22194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5517,"internal_metadata":27585139},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27056,"kb_used_data":96,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":94344784,"statfs":{"total":96636764160,"available":96609058816,"internally_reserved":0,"allocated":98304,"data_stored":22194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5519,"internal_metadata":27585137},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":420,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":430080,"data_stored":350386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5518,"internal_metadata":27585138},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":420,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":430080,"data_stored":350386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5519,"internal_metadata":27585137},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:54:48.283 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-12T03:54:48.283 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-12T03:54:48.283 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-12T03:54:48.284 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-12T03:54:48.496 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:48.496 INFO:teuthology.orchestra.run.trial103.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-12T03:54:48.505 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-12T03:54:48.505 INFO:teuthology.run_tasks:Running task backfill_toofull... 2026-02-12T03:54:48.524 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T03:54:48.703 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:48.703 INFO:teuthology.orchestra.run.trial103.stdout:{"epoch":12,"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","created":"2026-02-12T03:54:36.502189+0000","modified":"2026-02-12T03:54:46.669603+0000","last_up_change":"2026-02-12T03:54:41.512748+0000","last_in_change":"2026-02-12T03:54:38.267568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T03:54:43.668614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5abdfff6-6d88-4b2f-b2d5-4ef7df15e16f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6800","nonce":3386817029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6801","nonce":3386817029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6803","nonce":3386817029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6802","nonce":3386817029}]},"public_addr":"10.20.193.103:6800/3386817029","cluster_addr":"10.20.193.103:6801/3386817029","heartbeat_back_addr":"10.20.193.103:6803/3386817029","heartbeat_front_addr":"10.20.193.103:6802/3386817029","state":["exists","up"]},{"osd":1,"uuid":"619e1a4c-89fd-4b3f-89b2-59b67eff290c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6808","nonce":38137239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6810","nonce":38137239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6814","nonce":38137239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6812","nonce":38137239}]},"public_addr":"10.20.193.103:6808/38137239","cluster_addr":"10.20.193.103:6810/38137239","heartbeat_back_addr":"10.20.193.103:6814/38137239","heartbeat_front_addr":"10.20.193.103:6812/38137239","state":["exists","up"]},{"osd":2,"uuid":"d4795e0e-f032-4fa5-a993-f36b7e5b497f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6809","nonce":2960779923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6811","nonce":2960779923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6815","nonce":2960779923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6813","nonce":2960779923}]},"public_addr":"10.20.193.103:6809/2960779923","cluster_addr":"10.20.193.103:6811/2960779923","heartbeat_back_addr":"10.20.193.103:6815/2960779923","heartbeat_front_addr":"10.20.193.103:6813/2960779923","state":["exists","up"]},{"osd":3,"uuid":"4d822481-0e6c-42a6-b420-d6885a3f6cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6804","nonce":3177587099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6805","nonce":3177587099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6807","nonce":3177587099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6806","nonce":3177587099}]},"public_addr":"10.20.193.103:6804/3177587099","cluster_addr":"10.20.193.103:6805/3177587099","heartbeat_back_addr":"10.20.193.103:6807/3177587099","heartbeat_front_addr":"10.20.193.103:6806/3177587099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T03:54:48.713 INFO:tasks.backfill_toofull.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2026-02-12T03:54:43.668614+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'nonprimary_shards': '{}', 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.5, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2026-02-12T03:54:48.713 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2026-02-12T03:54:48.896 INFO:teuthology.orchestra.run.trial103.stdout:pg_num: 1 2026-02-12T03:54:48.904 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set backfill_toofull k=2 m=1 crush-failure-domain=osd 2026-02-12T03:54:49.682 INFO:teuthology.orchestra.run.trial103.stderr:load: isa 2026-02-12T03:54:49.694 INFO:tasks.backfill_toofull.ceph_manager:creating pool_name unique_pool_0 2026-02-12T03:54:49.695 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 1 1 erasure backfill_toofull 2026-02-12T03:54:51.688 INFO:teuthology.orchestra.run.trial103.stderr:pool 'unique_pool_0' created 2026-02-12T03:54:51.701 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2026-02-12T03:54:52.690 INFO:teuthology.orchestra.run.trial103.stderr:set pool 2 min_size to 2 2026-02-12T03:54:52.702 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2026-02-12T03:54:53.693 INFO:teuthology.orchestra.run.trial103.stderr:enabled application 'rados' on pool 'unique_pool_0' 2026-02-12T03:54:54.706 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_autoscale_mode off 2026-02-12T03:54:55.697 INFO:teuthology.orchestra.run.trial103.stderr:set pool 2 pg_autoscale_mode to off 2026-02-12T03:54:55.710 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-12T03:54:55.710 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-12T03:54:55.711 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-12T03:54:55.711 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-12T03:54:55.800 INFO:teuthology.orchestra.run.trial103.stdout:38654705669 2026-02-12T03:54:55.801 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-12T03:54:55.802 INFO:teuthology.orchestra.run.trial103.stdout:38654705669 2026-02-12T03:54:55.802 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-12T03:54:55.804 INFO:teuthology.orchestra.run.trial103.stdout:38654705669 2026-02-12T03:54:55.804 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-12T03:54:55.805 INFO:teuthology.orchestra.run.trial103.stdout:38654705669 2026-02-12T03:54:55.806 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-12T03:54:56.023 INFO:teuthology.orchestra.run.trial103.stdout:38654705668 2026-02-12T03:54:56.026 INFO:teuthology.orchestra.run.trial103.stdout:38654705668 2026-02-12T03:54:56.027 INFO:teuthology.orchestra.run.trial103.stdout:38654705668 2026-02-12T03:54:56.028 INFO:teuthology.orchestra.run.trial103.stdout:38654705668 2026-02-12T03:54:56.035 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705668 for osd.3 2026-02-12T03:54:56.037 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705668 for osd.1 2026-02-12T03:54:56.038 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705668 for osd.2 2026-02-12T03:54:56.039 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705668 for osd.0 2026-02-12T03:54:57.036 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-12T03:54:57.038 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-12T03:54:57.041 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-12T03:54:57.041 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-12T03:54:57.222 INFO:teuthology.orchestra.run.trial103.stdout:38654705668 2026-02-12T03:54:57.222 INFO:teuthology.orchestra.run.trial103.stdout:38654705668 2026-02-12T03:54:57.234 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705668 for osd.1 2026-02-12T03:54:57.235 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705668 for osd.3 2026-02-12T03:54:57.261 INFO:teuthology.orchestra.run.trial103.stdout:38654705668 2026-02-12T03:54:57.264 INFO:teuthology.orchestra.run.trial103.stdout:38654705668 2026-02-12T03:54:57.272 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705668 for osd.0 2026-02-12T03:54:57.275 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705668 for osd.2 2026-02-12T03:54:58.235 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-12T03:54:58.236 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-12T03:54:58.273 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-12T03:54:58.276 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-12T03:54:58.420 INFO:teuthology.orchestra.run.trial103.stdout:38654705669 2026-02-12T03:54:58.422 INFO:teuthology.orchestra.run.trial103.stdout:38654705669 2026-02-12T03:54:58.431 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705669 for osd.3 2026-02-12T03:54:58.432 DEBUG:teuthology.parallel:result is None 2026-02-12T03:54:58.435 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705669 for osd.1 2026-02-12T03:54:58.436 DEBUG:teuthology.parallel:result is None 2026-02-12T03:54:58.462 INFO:teuthology.orchestra.run.trial103.stdout:38654705669 2026-02-12T03:54:58.464 INFO:teuthology.orchestra.run.trial103.stdout:38654705669 2026-02-12T03:54:58.473 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705669 for osd.0 2026-02-12T03:54:58.474 DEBUG:teuthology.parallel:result is None 2026-02-12T03:54:58.475 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705669 for osd.2 2026-02-12T03:54:58.475 DEBUG:teuthology.parallel:result is None 2026-02-12T03:54:58.475 INFO:tasks.backfill_toofull.ceph_manager:waiting for clean 2026-02-12T03:54:58.475 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:54:58.693 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:58.693 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:54:58.703 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2026-02-12T03:54:57.658291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":109064,"kb_used_data":1224,"kb_used_omap":36,"kb_used_meta":107739,"kb_avail":377378296,"statfs":{"total":386547056640,"available":386435375104,"internally_reserved":0,"allocated":1253376,"data_stored":829752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37047,"internal_metadata":110325577},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.969619"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T03:54:55.697728+0000","last_change":"2026-02-12T03:54:53.393880+0000","last_active":"2026-02-12T03:54:55.697728+0000","last_peered":"2026-02-12T03:54:55.697728+0000","last_clean":"2026-02-12T03:54:55.697728+0000","last_became_active":"2026-02-12T03:54:53.393710+0000","last_became_peered":"2026-02-12T03:54:53.393710+0000","last_unstale":"2026-02-12T03:54:55.697728+0000","last_undegraded":"2026-02-12T03:54:55.697728+0000","last_fullsized":"2026-02-12T03:54:55.697728+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T11:55:11.608512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":47,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T03:54:55.697592+0000","last_change":"2026-02-12T03:54:46.118818+0000","last_active":"2026-02-12T03:54:55.697592+0000","last_peered":"2026-02-12T03:54:55.697592+0000","last_clean":"2026-02-12T03:54:55.697592+0000","last_became_active":"2026-02-12T03:54:46.118306+0000","last_became_peered":"2026-02-12T03:54:46.118306+0000","last_unstale":"2026-02-12T03:54:55.697592+0000","last_undegraded":"2026-02-12T03:54:55.697592+0000","last_fullsized":"2026-02-12T03:54:55.697592+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T13:21:54.271371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27104,"kb_used_data":144,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344736,"statfs":{"total":96636764160,"available":96609009664,"internally_reserved":0,"allocated":147456,"data_stored":43342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9423,"internal_metadata":27581233},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27104,"kb_used_data":144,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344736,"statfs":{"total":96636764160,"available":96609009664,"internally_reserved":0,"allocated":147456,"data_stored":43342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9425,"internal_metadata":27581231},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":468,"kb_used_omap":8,"kb_used_meta":26935,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":479232,"data_stored":371534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8774,"internal_metadata":27581882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":468,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":479232,"data_stored":371534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9425,"internal_metadata":27581231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:54:58.703 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:54:58.878 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:58.879 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:54:58.887 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2026-02-12T03:54:57.658291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":109064,"kb_used_data":1224,"kb_used_omap":36,"kb_used_meta":107739,"kb_avail":377378296,"statfs":{"total":386547056640,"available":386435375104,"internally_reserved":0,"allocated":1253376,"data_stored":829752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37047,"internal_metadata":110325577},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.969619"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T03:54:55.697728+0000","last_change":"2026-02-12T03:54:53.393880+0000","last_active":"2026-02-12T03:54:55.697728+0000","last_peered":"2026-02-12T03:54:55.697728+0000","last_clean":"2026-02-12T03:54:55.697728+0000","last_became_active":"2026-02-12T03:54:53.393710+0000","last_became_peered":"2026-02-12T03:54:53.393710+0000","last_unstale":"2026-02-12T03:54:55.697728+0000","last_undegraded":"2026-02-12T03:54:55.697728+0000","last_fullsized":"2026-02-12T03:54:55.697728+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T11:55:11.608512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":47,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T03:54:55.697592+0000","last_change":"2026-02-12T03:54:46.118818+0000","last_active":"2026-02-12T03:54:55.697592+0000","last_peered":"2026-02-12T03:54:55.697592+0000","last_clean":"2026-02-12T03:54:55.697592+0000","last_became_active":"2026-02-12T03:54:46.118306+0000","last_became_peered":"2026-02-12T03:54:46.118306+0000","last_unstale":"2026-02-12T03:54:55.697592+0000","last_undegraded":"2026-02-12T03:54:55.697592+0000","last_fullsized":"2026-02-12T03:54:55.697592+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T13:21:54.271371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27104,"kb_used_data":144,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344736,"statfs":{"total":96636764160,"available":96609009664,"internally_reserved":0,"allocated":147456,"data_stored":43342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9423,"internal_metadata":27581233},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27104,"kb_used_data":144,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344736,"statfs":{"total":96636764160,"available":96609009664,"internally_reserved":0,"allocated":147456,"data_stored":43342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9425,"internal_metadata":27581231},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":468,"kb_used_omap":8,"kb_used_meta":26935,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":479232,"data_stored":371534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8774,"internal_metadata":27581882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":468,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":479232,"data_stored":371534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9425,"internal_metadata":27581231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:54:58.888 INFO:tasks.backfill_toofull.ceph_manager:clean! 2026-02-12T03:54:58.888 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T03:54:59.072 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:59.072 INFO:teuthology.orchestra.run.trial103.stdout:{"epoch":18,"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","created":"2026-02-12T03:54:36.502189+0000","modified":"2026-02-12T03:54:55.694411+0000","last_up_change":"2026-02-12T03:54:41.512748+0000","last_in_change":"2026-02-12T03:54:38.267568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T03:54:43.668614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-12T03:54:50.692009+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_toofull","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"5abdfff6-6d88-4b2f-b2d5-4ef7df15e16f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6800","nonce":3386817029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6801","nonce":3386817029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6803","nonce":3386817029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6802","nonce":3386817029}]},"public_addr":"10.20.193.103:6800/3386817029","cluster_addr":"10.20.193.103:6801/3386817029","heartbeat_back_addr":"10.20.193.103:6803/3386817029","heartbeat_front_addr":"10.20.193.103:6802/3386817029","state":["exists","up"]},{"osd":1,"uuid":"619e1a4c-89fd-4b3f-89b2-59b67eff290c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6808","nonce":38137239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6810","nonce":38137239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6814","nonce":38137239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6812","nonce":38137239}]},"public_addr":"10.20.193.103:6808/38137239","cluster_addr":"10.20.193.103:6810/38137239","heartbeat_back_addr":"10.20.193.103:6814/38137239","heartbeat_front_addr":"10.20.193.103:6812/38137239","state":["exists","up"]},{"osd":2,"uuid":"d4795e0e-f032-4fa5-a993-f36b7e5b497f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6809","nonce":2960779923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6811","nonce":2960779923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6815","nonce":2960779923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6813","nonce":2960779923}]},"public_addr":"10.20.193.103:6809/2960779923","cluster_addr":"10.20.193.103:6811/2960779923","heartbeat_back_addr":"10.20.193.103:6815/2960779923","heartbeat_front_addr":"10.20.193.103:6813/2960779923","state":["exists","up"]},{"osd":3,"uuid":"4d822481-0e6c-42a6-b420-d6885a3f6cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6804","nonce":3177587099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6805","nonce":3177587099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6807","nonce":3177587099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6806","nonce":3177587099}]},"public_addr":"10.20.193.103:6804/3177587099","cluster_addr":"10.20.193.103:6805/3177587099","heartbeat_back_addr":"10.20.193.103:6807/3177587099","heartbeat_front_addr":"10.20.193.103:6806/3177587099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_toofull":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T03:54:59.083 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:54:59.263 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:54:59.263 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:54:59.272 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2026-02-12T03:54:57.658291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":109064,"kb_used_data":1224,"kb_used_omap":36,"kb_used_meta":107739,"kb_avail":377378296,"statfs":{"total":386547056640,"available":386435375104,"internally_reserved":0,"allocated":1253376,"data_stored":829752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37047,"internal_metadata":110325577},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.969619"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T03:54:55.697728+0000","last_change":"2026-02-12T03:54:53.393880+0000","last_active":"2026-02-12T03:54:55.697728+0000","last_peered":"2026-02-12T03:54:55.697728+0000","last_clean":"2026-02-12T03:54:55.697728+0000","last_became_active":"2026-02-12T03:54:53.393710+0000","last_became_peered":"2026-02-12T03:54:53.393710+0000","last_unstale":"2026-02-12T03:54:55.697728+0000","last_undegraded":"2026-02-12T03:54:55.697728+0000","last_fullsized":"2026-02-12T03:54:55.697728+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T11:55:11.608512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":47,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T03:54:55.697592+0000","last_change":"2026-02-12T03:54:46.118818+0000","last_active":"2026-02-12T03:54:55.697592+0000","last_peered":"2026-02-12T03:54:55.697592+0000","last_clean":"2026-02-12T03:54:55.697592+0000","last_became_active":"2026-02-12T03:54:46.118306+0000","last_became_peered":"2026-02-12T03:54:46.118306+0000","last_unstale":"2026-02-12T03:54:55.697592+0000","last_undegraded":"2026-02-12T03:54:55.697592+0000","last_fullsized":"2026-02-12T03:54:55.697592+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T13:21:54.271371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27104,"kb_used_data":144,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344736,"statfs":{"total":96636764160,"available":96609009664,"internally_reserved":0,"allocated":147456,"data_stored":43342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9423,"internal_metadata":27581233},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27104,"kb_used_data":144,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344736,"statfs":{"total":96636764160,"available":96609009664,"internally_reserved":0,"allocated":147456,"data_stored":43342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9425,"internal_metadata":27581231},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":468,"kb_used_omap":8,"kb_used_meta":26935,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":479232,"data_stored":371534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8774,"internal_metadata":27581882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":468,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":479232,"data_stored":371534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9425,"internal_metadata":27581231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:54:59.273 DEBUG:tasks.backfill_toofull:acting=[3, 1, 2] 2026-02-12T03:54:59.273 DEBUG:tasks.backfill_toofull:write some data 2026-02-12T03:54:59.273 INFO:tasks.util.rados:rados -p unique_pool_0 bench 120 write --no-cleanup 2026-02-12T03:54:59.273 DEBUG:teuthology.orchestra.run.trial103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 bench 120 write --no-cleanup 2026-02-12T03:54:59.310 INFO:teuthology.orchestra.run.trial103.stdout:hints = 1 2026-02-12T03:54:59.311 INFO:teuthology.orchestra.run.trial103.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 120 seconds or 0 objects 2026-02-12T03:54:59.311 INFO:teuthology.orchestra.run.trial103.stdout:Object prefix: benchmark_data_trial103_32382 2026-02-12T03:54:59.330 INFO:teuthology.orchestra.run.trial103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-12T03:54:59.330 INFO:teuthology.orchestra.run.trial103.stdout: 0 0 0 0 0 0 - 0 2026-02-12T03:54:59.788 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:54:59.787+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x55840f1641d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:00.045 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:00.044+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:00.330 INFO:teuthology.orchestra.run.trial103.stdout: 1 16 215 199 795.928 796 0.0524863 0.0784366 2026-02-12T03:55:00.427 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:00.426+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:00.427 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:00.426+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:00.633 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:00.632+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:00.843 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:00.842+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f1000083c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:01.331 INFO:teuthology.orchestra.run.trial103.stdout: 2 16 430 414 827.936 860 0.0463835 0.0724273 2026-02-12T03:55:02.330 INFO:teuthology.orchestra.run.trial103.stdout: 3 16 666 650 866.601 944 0.0478587 0.068143 2026-02-12T03:55:03.331 INFO:teuthology.orchestra.run.trial103.stdout: 4 16 930 914 913.933 1056 0.0525148 0.0689575 2026-02-12T03:55:03.548 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:03.548+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:03.972 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:03.972+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:03.986 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:03.985+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:04.258 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:04.257+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:04.331 INFO:teuthology.orchestra.run.trial103.stdout: 5 16 1206 1190 951.931 1104 0.0437328 0.0669296 2026-02-12T03:55:05.073 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:05.072+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:05.074 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:05.073+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:05.331 INFO:teuthology.orchestra.run.trial103.stdout: 6 16 1451 1435 956.598 980 0.266352 0.0653711 2026-02-12T03:55:05.547 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:05.546+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:06.284 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:06.284+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:06.331 INFO:teuthology.orchestra.run.trial103.stdout: 7 16 1658 1642 938.218 828 0.0413488 0.0679235 2026-02-12T03:55:07.165 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:07.164+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:07.331 INFO:teuthology.orchestra.run.trial103.stdout: 8 16 1976 1960 979.929 1272 0.0438252 0.065111 2026-02-12T03:55:07.456 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:07.455+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:08.331 INFO:teuthology.orchestra.run.trial103.stdout: 9 16 2231 2215 984.372 1020 0.0517648 0.0647994 2026-02-12T03:55:08.338 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:08.338+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:08.449 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:08.448+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:08.539 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:08.538+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:08.757 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:08.757+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:09.331 INFO:teuthology.orchestra.run.trial103.stdout: 10 16 2502 2486 994.328 1084 0.0467126 0.0642168 2026-02-12T03:55:09.897 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:09.895+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:10.245 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:10.243+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:10.331 INFO:teuthology.orchestra.run.trial103.stdout: 11 16 2670 2654 965.02 672 0.0505059 0.0656634 2026-02-12T03:55:11.305 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:11.304+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:11.327 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:11.325+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:11.331 INFO:teuthology.orchestra.run.trial103.stdout: 12 16 2942 2926 975.261 1088 0.0572373 0.0654513 2026-02-12T03:55:11.356 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:11.354+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:11.390 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:11.389+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:11.679 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:11.677+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:12.327 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:12.325+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:12.331 INFO:teuthology.orchestra.run.trial103.stdout: 13 16 3263 3247 999.004 1284 0.0477815 0.0639111 2026-02-12T03:55:12.417 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:12.415+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:12.665 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:12.663+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:12.742 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:12.740+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:13.331 INFO:teuthology.orchestra.run.trial103.stdout: 14 16 3445 3429 979.643 728 0.0519805 0.0652034 2026-02-12T03:55:13.357 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:13.356+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:13.442 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:13.440+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:13.453 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:13.451+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:13.905 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:13.903+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:14.045 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:14.043+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:14.332 INFO:teuthology.orchestra.run.trial103.stdout: 15 16 3760 3744 998.327 1260 0.0516213 0.0639854 2026-02-12T03:55:15.331 INFO:teuthology.orchestra.run.trial103.stdout: 16 16 4017 4001 1000.18 1028 0.245457 0.0632663 2026-02-12T03:55:15.553 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:15.551+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:15.905 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:15.904+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:15.930 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:15.928+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:16.332 INFO:teuthology.orchestra.run.trial103.stdout: 17 16 4348 4332 1019.22 1324 0.0558264 0.0626899 2026-02-12T03:55:17.185 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:17.184+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:17.331 INFO:teuthology.orchestra.run.trial103.stdout: 18 16 4589 4573 1016.15 964 0.0471595 0.0628979 2026-02-12T03:55:17.351 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:17.349+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:17.901 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:17.899+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:18.332 INFO:teuthology.orchestra.run.trial103.stdout: 19 16 4848 4832 1017.19 1036 0.0475813 0.0622764 2026-02-12T03:55:18.508 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:18.506+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:18.565 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:18.564+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:19.332 INFO:teuthology.orchestra.run.trial103.stdout:2026-02-12T03:55:19.332150+0000 min lat: 0.0111768 max lat: 0.468325 avg lat: 0.0621781 2026-02-12T03:55:19.332 INFO:teuthology.orchestra.run.trial103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-12T03:55:19.332 INFO:teuthology.orchestra.run.trial103.stdout: 20 16 5153 5137 1027.33 1220 0.0545679 0.0621781 2026-02-12T03:55:20.332 INFO:teuthology.orchestra.run.trial103.stdout: 21 16 5418 5402 1028.88 1060 0.0474382 0.0621304 2026-02-12T03:55:21.332 INFO:teuthology.orchestra.run.trial103.stdout: 22 16 5684 5668 1030.47 1064 0.050009 0.0620318 2026-02-12T03:55:22.332 INFO:teuthology.orchestra.run.trial103.stdout: 23 16 5894 5878 1022.19 840 0.0515755 0.0625391 2026-02-12T03:55:22.403 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:22.401+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:23.332 INFO:teuthology.orchestra.run.trial103.stdout: 24 16 6209 6193 1032.09 1260 0.0495908 0.0619276 2026-02-12T03:55:24.332 INFO:teuthology.orchestra.run.trial103.stdout: 25 16 6393 6377 1020.25 736 0.0584195 0.0615758 2026-02-12T03:55:25.332 INFO:teuthology.orchestra.run.trial103.stdout: 26 16 6393 6377 981.008 0 - 0.0615758 2026-02-12T03:55:26.332 INFO:teuthology.orchestra.run.trial103.stdout: 27 16 6393 6377 944.676 0 - 0.0615758 2026-02-12T03:55:27.332 INFO:teuthology.orchestra.run.trial103.stdout: 28 16 6393 6377 910.938 0 - 0.0615758 2026-02-12T03:55:28.332 INFO:teuthology.orchestra.run.trial103.stdout: 29 16 6393 6377 879.528 0 - 0.0615758 2026-02-12T03:55:29.332 INFO:teuthology.orchestra.run.trial103.stdout: 30 16 6393 6377 850.212 0 - 0.0615758 2026-02-12T03:55:30.332 INFO:teuthology.orchestra.run.trial103.stdout: 31 16 6393 6377 822.787 0 - 0.0615758 2026-02-12T03:55:31.332 INFO:teuthology.orchestra.run.trial103.stdout: 32 16 6393 6377 797.076 0 - 0.0615758 2026-02-12T03:55:32.332 INFO:teuthology.orchestra.run.trial103.stdout: 33 16 6393 6377 772.923 0 - 0.0615758 2026-02-12T03:55:33.332 INFO:teuthology.orchestra.run.trial103.stdout: 34 16 6393 6377 750.191 0 - 0.0615758 2026-02-12T03:55:34.333 INFO:teuthology.orchestra.run.trial103.stdout: 35 16 6393 6377 728.757 0 - 0.0615758 2026-02-12T03:55:35.333 INFO:teuthology.orchestra.run.trial103.stdout: 36 16 6393 6377 708.514 0 - 0.0615758 2026-02-12T03:55:36.333 INFO:teuthology.orchestra.run.trial103.stdout: 37 16 6393 6377 689.366 0 - 0.0615758 2026-02-12T03:55:37.333 INFO:teuthology.orchestra.run.trial103.stdout: 38 16 6393 6377 671.225 0 - 0.0615758 2026-02-12T03:55:38.333 INFO:teuthology.orchestra.run.trial103.stdout: 39 16 6393 6377 654.015 0 - 0.0615758 2026-02-12T03:55:39.130 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:39.129+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:39.333 INFO:teuthology.orchestra.run.trial103.stdout:2026-02-12T03:55:39.332964+0000 min lat: 0.0111768 max lat: 15.0649 avg lat: 0.0985678 2026-02-12T03:55:39.333 INFO:teuthology.orchestra.run.trial103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-12T03:55:39.333 INFO:teuthology.orchestra.run.trial103.stdout: 40 16 6473 6457 645.664 21.3333 0.0465599 0.0985678 2026-02-12T03:55:39.427 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:39.426+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:39.530 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:39.529+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:40.054 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:40.053+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:40.333 INFO:teuthology.orchestra.run.trial103.stdout: 41 16 6783 6767 660.158 1240 0.0536234 0.0968645 2026-02-12T03:55:40.738 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:40.737+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:40.787 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:40.786+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:41.250 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:41.249+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:41.271 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:41.270+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:41.321 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:41.320+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:41.333 INFO:teuthology.orchestra.run.trial103.stdout: 42 16 7050 7034 669.867 1068 0.0587505 0.0954699 2026-02-12T03:55:42.333 INFO:teuthology.orchestra.run.trial103.stdout: 43 16 7312 7296 678.659 1048 0.0532916 0.0942351 2026-02-12T03:55:43.246 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:43.245+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:43.333 INFO:teuthology.orchestra.run.trial103.stdout: 44 16 7568 7552 686.506 1024 0.0569861 0.0931519 2026-02-12T03:55:43.351 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:43.350+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:44.333 INFO:teuthology.orchestra.run.trial103.stdout: 45 16 7777 7761 689.827 836 0.0467567 0.0927237 2026-02-12T03:55:44.889 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:44.887+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:45.333 INFO:teuthology.orchestra.run.trial103.stdout: 46 16 8097 8081 702.655 1280 0.0492996 0.0910331 2026-02-12T03:55:45.431 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:45.429+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:45.951 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:45.949+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:46.333 INFO:teuthology.orchestra.run.trial103.stdout: 47 16 8416 8400 714.852 1276 0.0523847 0.0894786 2026-02-12T03:55:46.586 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:46.584+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:46.854 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:46.852+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:46.943 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:46.940+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:47.333 INFO:teuthology.orchestra.run.trial103.stdout: 48 16 8724 8708 725.624 1232 0.0521082 0.0881456 2026-02-12T03:55:48.010 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:48.008+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:48.245 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:48.243+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:48.267 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:48.265+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:48.333 INFO:teuthology.orchestra.run.trial103.stdout: 49 16 9040 9024 736.609 1264 0.0421354 0.0868415 2026-02-12T03:55:48.398 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:48.396+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:48.924 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:48.922+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:49.333 INFO:teuthology.orchestra.run.trial103.stdout: 50 16 9355 9339 747.075 1260 0.0455991 0.0856264 2026-02-12T03:55:49.792 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:49.790+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:50.333 INFO:teuthology.orchestra.run.trial103.stdout: 51 16 9668 9652 756.974 1252 0.0513666 0.084497 2026-02-12T03:55:50.993 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:50.991+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:51.085 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:51.083+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:51.334 INFO:teuthology.orchestra.run.trial103.stdout: 52 16 9917 9901 761.569 996 0.252132 0.0839526 2026-02-12T03:55:51.498 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:51.496+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:52.334 INFO:teuthology.orchestra.run.trial103.stdout: 53 16 10154 10138 765.086 948 0.0481185 0.0835239 2026-02-12T03:55:53.334 INFO:teuthology.orchestra.run.trial103.stdout: 54 16 10304 10288 762.028 600 0.0555906 0.0839407 2026-02-12T03:55:53.380 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:53.378+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:53.433 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:53.431+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:54.285 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:54.283+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:54.334 INFO:teuthology.orchestra.run.trial103.stdout: 55 16 10608 10592 770.281 1216 0.0619889 0.0830428 2026-02-12T03:55:55.334 INFO:teuthology.orchestra.run.trial103.stdout: 56 16 10919 10903 778.738 1244 0.0521199 0.082144 2026-02-12T03:55:56.334 INFO:teuthology.orchestra.run.trial103.stdout: 57 16 11158 11142 781.847 956 0.0487574 0.0815354 2026-02-12T03:55:57.334 INFO:teuthology.orchestra.run.trial103.stdout: 58 16 11364 11348 782.573 824 0.025473 0.081757 2026-02-12T03:55:57.415 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:57.413+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:57.449 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:57.447+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:58.326 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:58.324+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:58.327 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:58.324+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:55:58.334 INFO:teuthology.orchestra.run.trial103.stdout: 59 16 11606 11590 785.715 968 0.0498274 0.0814068 2026-02-12T03:55:58.676 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:58.674+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:58.863 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:58.861+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:59.267 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:59.265+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:59.334 INFO:teuthology.orchestra.run.trial103.stdout:2026-02-12T03:55:59.334398+0000 min lat: 0.0111768 max lat: 15.0649 avg lat: 0.0814694 2026-02-12T03:55:59.334 INFO:teuthology.orchestra.run.trial103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-12T03:55:59.334 INFO:teuthology.orchestra.run.trial103.stdout: 60 16 11795 11779 785.219 756 0.0471374 0.0814694 2026-02-12T03:55:59.339 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:59.338+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:55:59.635 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:55:59.633+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:00.334 INFO:teuthology.orchestra.run.trial103.stdout: 61 16 12043 12027 788.607 992 0.0503889 0.0811151 2026-02-12T03:56:00.639 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:00.637+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:01.334 INFO:teuthology.orchestra.run.trial103.stdout: 62 16 12362 12346 796.467 1276 0.0438742 0.0803272 2026-02-12T03:56:02.335 INFO:teuthology.orchestra.run.trial103.stdout: 63 16 12617 12601 800.014 1020 0.0477098 0.0799643 2026-02-12T03:56:03.334 INFO:teuthology.orchestra.run.trial103.stdout: 64 16 12927 12911 806.888 1240 0.0531734 0.0792788 2026-02-12T03:56:04.236 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:04.235+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:04.335 INFO:teuthology.orchestra.run.trial103.stdout: 65 16 13232 13216 813.242 1220 0.0516321 0.0786599 2026-02-12T03:56:04.586 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:04.585+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:05.335 INFO:teuthology.orchestra.run.trial103.stdout: 66 16 13387 13371 810.313 620 0.0530223 0.0787806 2026-02-12T03:56:05.871 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:05.870+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:06.203 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:06.202+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:06.335 INFO:teuthology.orchestra.run.trial103.stdout: 67 16 13684 13668 815.949 1188 0.0532591 0.0784021 2026-02-12T03:56:06.459 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:06.458+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:06.580 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:06.579+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:07.135 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:07.134+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:07.303 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:07.302+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:07.335 INFO:teuthology.orchestra.run.trial103.stdout: 68 16 13976 13960 821.125 1168 0.0549135 0.0778949 2026-02-12T03:56:07.842 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:07.841+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:07.909 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:07.908+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:07.936 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:07.935+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:08.335 INFO:teuthology.orchestra.run.trial103.stdout: 69 16 14167 14151 820.296 764 0.050207 0.0779891 2026-02-12T03:56:08.640 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:08.639+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:08.710 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:08.709+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:09.199 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:09.198+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:09.335 INFO:teuthology.orchestra.run.trial103.stdout: 70 16 14420 14404 823.034 1012 0.0509957 0.0777262 2026-02-12T03:56:10.189 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:10.188+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:10.245 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:10.244+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:10.335 INFO:teuthology.orchestra.run.trial103.stdout: 71 16 14689 14673 826.596 1076 0.0495897 0.0773937 2026-02-12T03:56:10.627 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:10.626+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x55840f143d40 msgr2=0x7f10000811e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:11.335 INFO:teuthology.orchestra.run.trial103.stdout: 72 16 14993 14977 832.003 1216 0.0523782 0.0768875 2026-02-12T03:56:12.116 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:12.115+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f1000081820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:12.335 INFO:teuthology.orchestra.run.trial103.stdout: 73 16 15239 15223 834.084 984 0.0519687 0.0766997 2026-02-12T03:56:13.335 INFO:teuthology.orchestra.run.trial103.stdout: 74 16 15497 15481 836.758 1032 0.0489366 0.0764529 2026-02-12T03:56:13.361 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:13.360+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000843d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:14.335 INFO:teuthology.orchestra.run.trial103.stdout: 75 16 15637 15621 833.067 560 0.0498216 0.076796 2026-02-12T03:56:14.460 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:14.459+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:15.214 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:15.213+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f1000083740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:15.265 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:15.265+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:15.335 INFO:teuthology.orchestra.run.trial103.stdout: 76 16 15890 15874 835.42 1012 0.0402448 0.076582 2026-02-12T03:56:15.594 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:15.593+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f1000081820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:16.022 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:16.021+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:16.335 INFO:teuthology.orchestra.run.trial103.stdout: 77 16 16150 16134 838.076 1040 0.0504037 0.076332 2026-02-12T03:56:16.472 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:16.471+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f1000081820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:16.625 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:16.624+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:17.004 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:17.003+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f1000081820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:17.335 INFO:teuthology.orchestra.run.trial103.stdout: 78 16 16391 16375 839.69 964 0.0487157 0.0759977 2026-02-12T03:56:17.577 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:17.576+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:17.780 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:17.780+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:18.043 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:18.042+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000833c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:18.044 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:18.043+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:18.336 INFO:teuthology.orchestra.run.trial103.stdout: 79 16 16704 16688 844.908 1252 0.0485216 0.0757237 2026-02-12T03:56:18.354 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:18.353+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f1000081820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:18.423 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:18.422+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:19.336 INFO:teuthology.orchestra.run.trial103.stdout:2026-02-12T03:56:19.335880+0000 min lat: 0.0100983 max lat: 15.0649 avg lat: 0.0752667 2026-02-12T03:56:19.336 INFO:teuthology.orchestra.run.trial103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-12T03:56:19.336 INFO:teuthology.orchestra.run.trial103.stdout: 80 16 17017 17001 849.995 1252 0.0502629 0.0752667 2026-02-12T03:56:20.336 INFO:teuthology.orchestra.run.trial103.stdout: 81 16 17124 17108 844.785 428 0.055989 0.0752577 2026-02-12T03:56:20.461 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:20.460+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f1000081760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:20.461 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:20.460+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:20.914 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:20.913+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:21.068 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:21.067+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000822a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:21.078 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:21.078+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:21.336 INFO:teuthology.orchestra.run.trial103.stdout: 82 16 17338 17322 844.921 856 0.0527867 0.0757191 2026-02-12T03:56:22.080 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:22.079+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:22.300 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:22.299+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:22.336 INFO:teuthology.orchestra.run.trial103.stdout: 83 16 17638 17622 849.198 1200 0.0623258 0.075337 2026-02-12T03:56:22.438 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:22.437+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:22.748 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:22.747+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:22.819 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:22.818+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:22.939 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:22.938+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:22.955 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:22.954+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:22.972 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:22.971+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:23.254 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:23.253+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000815b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:23.336 INFO:teuthology.orchestra.run.trial103.stdout: 84 16 17865 17849 849.897 908 0.0498297 0.0752787 2026-02-12T03:56:23.794 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:23.793+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:24.093 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:24.092+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:24.336 INFO:teuthology.orchestra.run.trial103.stdout: 85 16 18163 18147 853.921 1192 0.0650792 0.0749151 2026-02-12T03:56:25.336 INFO:teuthology.orchestra.run.trial103.stdout: 86 16 18468 18452 858.176 1220 0.0582146 0.0745456 2026-02-12T03:56:25.569 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:25.569+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:25.931 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:25.931+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:26.266 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:26.265+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:26.336 INFO:teuthology.orchestra.run.trial103.stdout: 87 16 18707 18691 859.3 956 0.0492082 0.0743999 2026-02-12T03:56:26.935 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:26.934+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:26.936 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:26.935+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000828d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:27.336 INFO:teuthology.orchestra.run.trial103.stdout: 88 16 18904 18888 858.489 788 0.0486093 0.0745269 2026-02-12T03:56:27.460 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:27.459+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:27.537 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:27.537+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000831d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:27.684 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:27.683+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:28.103 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:28.103+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000834d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:28.336 INFO:teuthology.orchestra.run.trial103.stdout: 89 16 19158 19142 860.258 1016 0.0532752 0.0743722 2026-02-12T03:56:29.336 INFO:teuthology.orchestra.run.trial103.stdout: 90 16 19454 19438 863.854 1184 0.0537844 0.0740613 2026-02-12T03:56:29.846 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:29.845+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:29.856 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:29.855+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:30.336 INFO:teuthology.orchestra.run.trial103.stdout: 91 16 19663 19647 863.548 836 0.0490288 0.074006 2026-02-12T03:56:31.337 INFO:teuthology.orchestra.run.trial103.stdout: 92 16 19940 19924 866.204 1108 0.0560702 0.0738607 2026-02-12T03:56:32.337 INFO:teuthology.orchestra.run.trial103.stdout: 93 16 20252 20236 870.308 1248 0.0503599 0.073513 2026-02-12T03:56:32.393 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:32.392+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:33.337 INFO:teuthology.orchestra.run.trial103.stdout: 94 16 20562 20546 874.24 1240 0.0510449 0.073184 2026-02-12T03:56:34.337 INFO:teuthology.orchestra.run.trial103.stdout: 95 16 20870 20854 878.005 1232 0.0502005 0.072869 2026-02-12T03:56:34.344 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:34.343+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:34.360 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:34.358+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:34.498 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:34.496+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:34.499 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:34.497+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f1000082540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:34.970 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:34.968+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f1000082540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:35.280 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:35.278+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:35.337 INFO:teuthology.orchestra.run.trial103.stdout: 96 16 21058 21042 876.692 752 0.056922 0.0729785 2026-02-12T03:56:35.588 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:35.586+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:35.934 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:35.932+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:36.060 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:36.058+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:36.209 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:36.207+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:36.337 INFO:teuthology.orchestra.run.trial103.stdout: 97 16 21367 21351 880.395 1236 0.0532472 0.0726725 2026-02-12T03:56:36.421 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:36.419+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000828c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:36.838 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:36.836+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f1000082540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:37.243 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:37.241+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:37.337 INFO:teuthology.orchestra.run.trial103.stdout: 98 16 21596 21580 880.758 916 0.0474912 0.0725751 2026-02-12T03:56:37.605 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:37.603+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:38.337 INFO:teuthology.orchestra.run.trial103.stdout: 99 16 21811 21795 880.547 860 0.0531997 0.0726589 2026-02-12T03:56:38.434 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:38.432+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:38.601 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:38.599+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000815b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:38.667 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:38.665+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f1000084060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:39.337 INFO:teuthology.orchestra.run.trial103.stdout:2026-02-12T03:56:39.337416+0000 min lat: 0.0100983 max lat: 15.0649 avg lat: 0.0723802 2026-02-12T03:56:39.337 INFO:teuthology.orchestra.run.trial103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-12T03:56:39.338 INFO:teuthology.orchestra.run.trial103.stdout: 100 16 22115 22099 883.901 1216 0.0485353 0.0723802 2026-02-12T03:56:39.369 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:39.367+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f1000083c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:40.272 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:40.271+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:40.337 INFO:teuthology.orchestra.run.trial103.stdout: 101 16 22419 22403 887.188 1216 0.0416883 0.072118 2026-02-12T03:56:41.338 INFO:teuthology.orchestra.run.trial103.stdout: 102 16 22621 22605 886.411 808 0.0521807 0.0721742 2026-02-12T03:56:41.982 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:41.981+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:42.338 INFO:teuthology.orchestra.run.trial103.stdout: 103 16 22869 22853 887.435 992 0.0457692 0.0720935 2026-02-12T03:56:42.732 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:42.731+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:43.332 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:43.331+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:43.338 INFO:teuthology.orchestra.run.trial103.stdout: 104 15 23118 23103 888.517 1000 0.0488089 0.0720104 2026-02-12T03:56:43.730 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:43.729+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:44.115 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:44.113+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f1000082c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:44.338 INFO:teuthology.orchestra.run.trial103.stdout: 105 16 23420 23404 891.521 1204 0.046101 0.0717676 2026-02-12T03:56:44.701 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:44.699+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:45.338 INFO:teuthology.orchestra.run.trial103.stdout: 106 16 23681 23665 892.958 1044 0.0533381 0.0716495 2026-02-12T03:56:45.574 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:45.573+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:45.675 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:45.673+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:45.965 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:45.963+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:46.338 INFO:teuthology.orchestra.run.trial103.stdout: 107 16 24004 23988 896.687 1292 0.0475173 0.0713539 2026-02-12T03:56:47.338 INFO:teuthology.orchestra.run.trial103.stdout: 108 16 24260 24244 897.865 1024 0.0527677 0.0712588 2026-02-12T03:56:48.338 INFO:teuthology.orchestra.run.trial103.stdout: 109 16 24519 24503 899.132 1036 0.05437 0.0711575 2026-02-12T03:56:48.392 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:48.391+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:48.449 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:48.448+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:49.039 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:49.037+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:49.338 INFO:teuthology.orchestra.run.trial103.stdout: 110 16 24771 24755 900.121 1008 0.0238036 0.0710887 2026-02-12T03:56:50.338 INFO:teuthology.orchestra.run.trial103.stdout: 111 16 24994 24978 900.047 892 0.0434079 0.0709133 2026-02-12T03:56:51.062 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:51.060+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:51.338 INFO:teuthology.orchestra.run.trial103.stdout: 112 16 25219 25203 900.046 900 0.04593 0.0710897 2026-02-12T03:56:51.511 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:51.510+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:51.589 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:51.588+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:51.918 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:51.917+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:56:51.992 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:51.991+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:52.338 INFO:teuthology.orchestra.run.trial103.stdout: 113 16 25468 25452 900.894 996 0.0502643 0.0710221 2026-02-12T03:56:53.339 INFO:teuthology.orchestra.run.trial103.stdout: 114 16 25774 25758 903.728 1224 0.0533424 0.0707928 2026-02-12T03:56:53.484 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:53.483+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:53.761 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:53.760+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:53.928 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:53.927+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:54.339 INFO:teuthology.orchestra.run.trial103.stdout: 115 16 25989 25973 903.347 860 0.0499148 0.0707646 2026-02-12T03:56:54.444 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:54.443+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:54.516 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:54.515+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:54.794 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:54.793+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:55.339 INFO:teuthology.orchestra.run.trial103.stdout: 116 16 26273 26257 905.352 1136 0.0535678 0.0706648 2026-02-12T03:56:55.761 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:55.760+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:55.765 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:55.764+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000811e0 msgr2=0x7f10000a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:56.287 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:56.285+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:56.339 INFO:teuthology.orchestra.run.trial103.stdout: 117 16 26582 26566 908.177 1236 0.0541249 0.0704514 2026-02-12T03:56:56.801 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:56.800+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1000060560 msgr2=0x7f1000080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:57.150 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:57.149+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000843f0 msgr2=0x7f10000815b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:57.339 INFO:teuthology.orchestra.run.trial103.stdout: 118 16 26842 26826 909.294 1040 0.0460337 0.0703659 2026-02-12T03:56:58.339 INFO:teuthology.orchestra.run.trial103.stdout: 119 16 27148 27132 911.937 1224 0.0623302 0.0701562 2026-02-12T03:56:58.774 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:56:58.773+0000 7f1021f23640 0 -- 10.20.193.103:0/1269391554 >> v2:10.20.193.103:6804/3177587099 conn(0x7f10000843f0 msgr2=0x7f1000083bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:56:59.339 INFO:teuthology.orchestra.run.trial103.stdout:2026-02-12T03:56:59.338974+0000 min lat: 0.0100983 max lat: 15.0649 avg lat: 0.0700897 2026-02-12T03:56:59.339 INFO:teuthology.orchestra.run.trial103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-12T03:56:59.339 INFO:teuthology.orchestra.run.trial103.stdout: 120 12 27401 27389 912.904 1028 0.0483619 0.0700897 2026-02-12T03:57:00.339 INFO:teuthology.orchestra.run.trial103.stdout:Total time run: 120.038 2026-02-12T03:57:00.340 INFO:teuthology.orchestra.run.trial103.stdout:Total writes made: 27401 2026-02-12T03:57:00.340 INFO:teuthology.orchestra.run.trial103.stdout:Write size: 4194304 2026-02-12T03:57:00.340 INFO:teuthology.orchestra.run.trial103.stdout:Object size: 4194304 2026-02-12T03:57:00.340 INFO:teuthology.orchestra.run.trial103.stdout:Bandwidth (MB/sec): 913.074 2026-02-12T03:57:00.340 INFO:teuthology.orchestra.run.trial103.stdout:Stddev Bandwidth: 388.223 2026-02-12T03:57:00.340 INFO:teuthology.orchestra.run.trial103.stdout:Max bandwidth (MB/sec): 1324 2026-02-12T03:57:00.340 INFO:teuthology.orchestra.run.trial103.stdout:Min bandwidth (MB/sec): 0 2026-02-12T03:57:00.340 INFO:teuthology.orchestra.run.trial103.stdout:Average IOPS: 228 2026-02-12T03:57:00.340 INFO:teuthology.orchestra.run.trial103.stdout:Stddev IOPS: 97.0621 2026-02-12T03:57:00.340 INFO:teuthology.orchestra.run.trial103.stdout:Max IOPS: 331 2026-02-12T03:57:00.340 INFO:teuthology.orchestra.run.trial103.stdout:Min IOPS: 0 2026-02-12T03:57:00.340 INFO:teuthology.orchestra.run.trial103.stdout:Average Latency(s): 0.0700795 2026-02-12T03:57:00.341 INFO:teuthology.orchestra.run.trial103.stdout:Stddev Latency(s): 0.365537 2026-02-12T03:57:00.341 INFO:teuthology.orchestra.run.trial103.stdout:Max latency(s): 15.0649 2026-02-12T03:57:00.341 INFO:teuthology.orchestra.run.trial103.stdout:Min latency(s): 0.0100983 2026-02-12T03:57:00.349 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd df name osd.1 --format=json 2026-02-12T03:57:00.567 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:57:00.567 INFO:teuthology.orchestra.run.trial103.stdout:{"nodes":[{"id":1,"device_class":"ssd","name":"osd.1","type":"osd","type_id":0,"crush_weight":0.087890625,"depth":2,"pool_weights":{},"reweight":1,"kb":94371840,"kb_used":54990140,"kb_used_data":54800876,"kb_used_omap":10,"kb_used_meta":189237,"kb_avail":39381700,"utilization":58.269649081759979,"var":1,"pgs":2,"status":"up"}],"stray":[],"summary":{"total_kb":94371840,"total_kb_used":54990140,"total_kb_used_data":54800876,"total_kb_used_omap":10,"total_kb_used_meta":189237,"total_kb_avail":39381700,"average_utilization":58.269649081759979,"min_var":1,"max_var":1,"dev":0}} 2026-02-12T03:57:00.575 DEBUG:tasks.backfill_toofull:target osd df: {'id': 1, 'device_class': 'ssd', 'name': 'osd.1', 'type': 'osd', 'type_id': 0, 'crush_weight': 0.087890625, 'depth': 2, 'pool_weights': {}, 'reweight': 1, 'kb': 94371840, 'kb_used': 54990140, 'kb_used_data': 54800876, 'kb_used_omap': 10, 'kb_used_meta': 189237, 'kb_avail': 39381700, 'utilization': 58.26964908175998, 'var': 1, 'pgs': 2, 'status': 'up'} 2026-02-12T03:57:00.575 DEBUG:tasks.backfill_toofull:pause recovery 2026-02-12T03:57:00.575 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noout 2026-02-12T03:57:00.815 INFO:teuthology.orchestra.run.trial103.stderr:noout is set 2026-02-12T03:57:00.825 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nobackfill 2026-02-12T03:57:01.820 INFO:teuthology.orchestra.run.trial103.stderr:nobackfill is set 2026-02-12T03:57:01.831 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set norecover 2026-02-12T03:57:02.824 INFO:teuthology.orchestra.run.trial103.stderr:norecover is set 2026-02-12T03:57:02.835 DEBUG:tasks.backfill_toofull:stop tartget osd 1 2026-02-12T03:57:02.836 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-12T03:57:02.836 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T03:57:02.868 INFO:tasks.ceph.osd.1:Stopped 2026-02-12T03:57:02.868 INFO:tasks.backfill_toofull.ceph_manager:waiting till active 2026-02-12T03:57:02.869 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:57:03.087 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:57:03.087 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:57:03.096 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2026-02-12T03:57:02.823319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114928452120,"num_objects":27404,"num_object_clones":0,"num_object_copies":82210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27404,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":27439,"num_write_kb":112234819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":168947884,"kb_used_data":168353132,"kb_used_omap":48,"kb_used_meta":594639,"kb_avail":208539476,"statfs":{"total":386547056640,"available":213544423424,"internally_reserved":0,"allocated":172393607168,"data_stored":172393094276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":49448,"internal_metadata":608911064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7637827608,"num_objects":1822,"num_object_clones":0,"num_object_copies":5466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1822,"num_write_kb":7458817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.149948"},"pg_stats":[{"pgid":"2.0","version":"20'27402","reported_seq":27430,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-12T03:57:01.820154+0000","last_change":"2026-02-12T03:55:38.925931+0000","last_active":"2026-02-12T03:57:01.820154+0000","last_peered":"2026-02-12T03:57:01.820154+0000","last_clean":"2026-02-12T03:57:01.820154+0000","last_became_active":"2026-02-12T03:54:53.393710+0000","last_became_peered":"2026-02-12T03:54:53.393710+0000","last_unstale":"2026-02-12T03:57:01.820154+0000","last_undegraded":"2026-02-12T03:57:01.820154+0000","last_fullsized":"2026-02-12T03:57:01.820154+0000","mapping_epoch":15,"log_start":"20'27373","ondisk_log_start":"20'27373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T11:55:11.608512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114928123928,"num_objects":27402,"num_object_clones":0,"num_object_copies":82206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27402,"num_write_kb":112234497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-12T03:57:02.331006+0000","last_change":"2026-02-12T03:54:46.118818+0000","last_active":"2026-02-12T03:57:02.331006+0000","last_peered":"2026-02-12T03:57:02.331006+0000","last_clean":"2026-02-12T03:57:02.331006+0000","last_became_active":"2026-02-12T03:54:46.118306+0000","last_became_peered":"2026-02-12T03:54:46.118306+0000","last_unstale":"2026-02-12T03:57:02.331006+0000","last_undegraded":"2026-02-12T03:57:02.331006+0000","last_fullsized":"2026-02-12T03:57:02.331006+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T13:21:54.271371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":114928123928,"num_objects":27402,"num_object_clones":0,"num_object_copies":82206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27402,"num_write_kb":112234497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172392198144,"data_stored":172392198144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705695,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56306700,"kb_used_data":56117436,"kb_used_omap":12,"kb_used_meta":189235,"kb_avail":38065140,"statfs":{"total":96636764160,"available":38978703360,"internally_reserved":0,"allocated":57464254464,"data_stored":57464126829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12690,"internal_metadata":193777262},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56306700,"kb_used_data":56117436,"kb_used_omap":12,"kb_used_meta":189235,"kb_avail":38065140,"statfs":{"total":96636764160,"available":38978703360,"internally_reserved":0,"allocated":57464254464,"data_stored":57464126829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12692,"internal_metadata":193777260},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705695,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56307016,"kb_used_data":56117752,"kb_used_omap":11,"kb_used_meta":189236,"kb_avail":38064824,"statfs":{"total":96636764160,"available":38978379776,"internally_reserved":0,"allocated":57464578048,"data_stored":57464451645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11374,"internal_metadata":193778578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705695,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":508,"kb_used_omap":12,"kb_used_meta":26931,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":520192,"data_stored":388973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12692,"internal_metadata":27577964},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:57:03.097 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:57:03.324 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:57:03.324 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":146,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":23,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868481,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":2}],"num_pgs":2,"num_pools":2,"num_objects":26950,"data_bytes":113028432384,"bytes_used":168994906112,"bytes_avail":217552150528,"bytes_total":386547056640,"write_bytes_sec":1103789836,"read_op_per_sec":0,"write_op_per_sec":263},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":3,"modified":"2026-02-12T03:56:57.673729+0000","services":{}},"progress_events":{}} 2026-02-12T03:57:03.332 INFO:tasks.backfill_toofull.ceph_manager:{'fsid': 'e9633232-3262-4ad1-a60a-815554f29d9f', 'health': {'status': 'HEALTH_WARN', 'checks': {'OSDMAP_FLAGS': {'severity': 'HEALTH_WARN', 'summary': {'message': 'noout,nobackfill,norecover flag(s) set', 'count': 26}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 146, 'monmap': {'epoch': 1, 'min_mon_release_name': 'tentacle', 'num_mons': 3}, 'osdmap': {'epoch': 23, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1770868481, 'num_in_osds': 4, 'osd_in_since': 1770868478, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 2}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 26950, 'data_bytes': 113028432384, 'bytes_used': 168994906112, 'bytes_avail': 217552150528, 'bytes_total': 386547056640, 'write_bytes_sec': 1103789836, 'read_op_per_sec': 0, 'write_op_per_sec': 263}, 'fsmap': {'epoch': 1, 'btime': '2026-02-12T03:54:36:498858+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs'], 'services': {}}, 'servicemap': {'epoch': 3, 'modified': '2026-02-12T03:56:57.673729+0000', 'services': {}}, 'progress_events': {}} 2026-02-12T03:57:03.332 INFO:tasks.backfill_toofull.ceph_manager:active! 2026-02-12T03:57:03.332 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:57:03.509 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:57:03.510 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:57:03.518 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2026-02-12T03:57:02.823319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114928452120,"num_objects":27404,"num_object_clones":0,"num_object_copies":82210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27404,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":27439,"num_write_kb":112234819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":168947884,"kb_used_data":168353132,"kb_used_omap":48,"kb_used_meta":594639,"kb_avail":208539476,"statfs":{"total":386547056640,"available":213544423424,"internally_reserved":0,"allocated":172393607168,"data_stored":172393094276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":49448,"internal_metadata":608911064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7637827608,"num_objects":1822,"num_object_clones":0,"num_object_copies":5466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1822,"num_write_kb":7458817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.149948"},"pg_stats":[{"pgid":"2.0","version":"20'27402","reported_seq":27430,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-12T03:57:01.820154+0000","last_change":"2026-02-12T03:55:38.925931+0000","last_active":"2026-02-12T03:57:01.820154+0000","last_peered":"2026-02-12T03:57:01.820154+0000","last_clean":"2026-02-12T03:57:01.820154+0000","last_became_active":"2026-02-12T03:54:53.393710+0000","last_became_peered":"2026-02-12T03:54:53.393710+0000","last_unstale":"2026-02-12T03:57:01.820154+0000","last_undegraded":"2026-02-12T03:57:01.820154+0000","last_fullsized":"2026-02-12T03:57:01.820154+0000","mapping_epoch":15,"log_start":"20'27373","ondisk_log_start":"20'27373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T11:55:11.608512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114928123928,"num_objects":27402,"num_object_clones":0,"num_object_copies":82206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27402,"num_write_kb":112234497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-12T03:57:02.331006+0000","last_change":"2026-02-12T03:54:46.118818+0000","last_active":"2026-02-12T03:57:02.331006+0000","last_peered":"2026-02-12T03:57:02.331006+0000","last_clean":"2026-02-12T03:57:02.331006+0000","last_became_active":"2026-02-12T03:54:46.118306+0000","last_became_peered":"2026-02-12T03:54:46.118306+0000","last_unstale":"2026-02-12T03:57:02.331006+0000","last_undegraded":"2026-02-12T03:57:02.331006+0000","last_fullsized":"2026-02-12T03:57:02.331006+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T13:21:54.271371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":114928123928,"num_objects":27402,"num_object_clones":0,"num_object_copies":82206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27402,"num_write_kb":112234497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172392198144,"data_stored":172392198144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705695,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56306700,"kb_used_data":56117436,"kb_used_omap":12,"kb_used_meta":189235,"kb_avail":38065140,"statfs":{"total":96636764160,"available":38978703360,"internally_reserved":0,"allocated":57464254464,"data_stored":57464126829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12690,"internal_metadata":193777262},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56306700,"kb_used_data":56117436,"kb_used_omap":12,"kb_used_meta":189235,"kb_avail":38065140,"statfs":{"total":96636764160,"available":38978703360,"internally_reserved":0,"allocated":57464254464,"data_stored":57464126829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12692,"internal_metadata":193777260},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705695,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56307016,"kb_used_data":56117752,"kb_used_omap":11,"kb_used_meta":189236,"kb_avail":38064824,"statfs":{"total":96636764160,"available":38978379776,"internally_reserved":0,"allocated":57464578048,"data_stored":57464451645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11374,"internal_metadata":193778578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705695,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":508,"kb_used_omap":12,"kb_used_meta":26931,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":520192,"data_stored":388973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12692,"internal_metadata":27577964},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:57:03.519 DEBUG:tasks.backfill_toofull:pg={'pgid': '2.0', 'version': "20'27402", 'reported_seq': 27430, 'reported_epoch': 22, 'state': 'active+clean', 'last_fresh': '2026-02-12T03:57:01.820154+0000', 'last_change': '2026-02-12T03:55:38.925931+0000', 'last_active': '2026-02-12T03:57:01.820154+0000', 'last_peered': '2026-02-12T03:57:01.820154+0000', 'last_clean': '2026-02-12T03:57:01.820154+0000', 'last_became_active': '2026-02-12T03:54:53.393710+0000', 'last_became_peered': '2026-02-12T03:54:53.393710+0000', 'last_unstale': '2026-02-12T03:57:01.820154+0000', 'last_undegraded': '2026-02-12T03:57:01.820154+0000', 'last_fullsized': '2026-02-12T03:57:01.820154+0000', 'mapping_epoch': 15, 'log_start': "20'27373", 'ondisk_log_start': "20'27373", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 29, 'log_dups_size': 3000, 'ondisk_log_size': 29, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-13T11:55:11.608512+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 114928123928, 'num_objects': 27402, 'num_object_clones': 0, 'num_object_copies': 82206, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 27402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 27402, 'num_write_kb': 112234497, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:57:03.519 DEBUG:tasks.backfill_toofull:re-write data 2026-02-12T03:57:03.519 INFO:tasks.util.rados:rados -p unique_pool_0 cleanup 2026-02-12T03:57:03.519 DEBUG:teuthology.orchestra.run.trial103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 cleanup 2026-02-12T03:57:04.856 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:04.854+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x5556b57c80a0 msgr2=0x5556b57ae0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:04.865 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:04.863+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40605f0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:04.911 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:04.909+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4081790 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:04.911 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:04.909+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4081790 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:04.912 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:04.910+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x5556b57c80a0 msgr2=0x7f1fe4081210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:04.940 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:04.938+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x5556b57c80a0 msgr2=0x7f1fe4081210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:05.235 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:05.233+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x5556b57c80a0 msgr2=0x7f1fe4081210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:05.308 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:05.306+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x5556b57c80a0 msgr2=0x7f1fe4081210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:05.309 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:05.307+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:05.354 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:05.352+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:05.694 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:05.692+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:05.695 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:05.692+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4081210 msgr2=0x7f1fe4082b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:05.911 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:05.909+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4081210 msgr2=0x7f1fe4082b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:05.994 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:05.992+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:06.128 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:06.126+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:06.169 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:06.167+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:06.485 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:06.483+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:06.502 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:06.501+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082740 msgr2=0x7f1fe4082b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:06.514 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:06.512+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:06.548 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:06.546+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:06.663 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:06.661+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:06.720 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:06.718+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:06.861 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:06.859+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:06.865 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:06.863+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082740 msgr2=0x7f1fe4084500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:06.868 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:06.866+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:06.873 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:06.871+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082740 msgr2=0x7f1fe4084500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:06.921 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:06.919+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:07.030 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:07.028+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082740 msgr2=0x7f1fe4084500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:07.042 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:07.040+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:07.046 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:07.045+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:07.062 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:07.060+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082740 msgr2=0x7f1fe4084500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:07.130 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:07.128+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082740 msgr2=0x7f1fe4084500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:07.340 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:07.338+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:07.341 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:07.339+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082740 msgr2=0x7f1fe4084500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:07.411 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:07.409+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082740 msgr2=0x7f1fe4084500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:07.432 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:07.430+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:07.825 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:07.823+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082740 msgr2=0x7f1fe4084500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:07.893 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:07.891+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082740 msgr2=0x7f1fe4084500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:07.898 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:07.896+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:08.080 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:08.078+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4084500 msgr2=0x7f1fe4082740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:08.111 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:08.109+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:08.198 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:08.196+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:08.256 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:08.254+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:08.263 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:08.261+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4084500 msgr2=0x7f1fe4082740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:08.710 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:08.708+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4084500 msgr2=0x7f1fe4082740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:08.972 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:08.970+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082740 msgr2=0x7f1fe4082f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:08.990 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:08.988+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:09.014 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:09.012+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082740 msgr2=0x7f1fe40a52f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:09.090 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:09.088+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082740 msgr2=0x7f1fe40a52f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:09.172 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:09.170+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a6200 msgr2=0x7f1fe40a52f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:09.182 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:09.180+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:09.254 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:09.252+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a6200 msgr2=0x7f1fe40a65f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:09.460 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:09.458+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:09.465 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:09.463+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a6200 msgr2=0x7f1fe40a65f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:09.710 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:09.708+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:09.731 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:09.729+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a6200 msgr2=0x7f1fe40a65d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:09.906 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:09.904+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:09.982 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:09.979+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:10.066 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:10.064+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe4082370 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:10.152 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:10.150+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a6200 msgr2=0x7f1fe40a65d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:10.191 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:10.189+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a6200 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:10.204 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:10.202+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:10.395 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:10.392+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a6200 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:10.427 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:10.425+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:10.431 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:10.429+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a6200 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:10.454 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:10.452+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:10.565 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:10.563+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:10.703 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:10.701+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:10.869 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:10.867+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:10.883 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:10.881+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:10.911 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:10.909+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:11.078 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:11.076+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:11.085 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:11.083+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:11.213 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:11.211+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:11.625 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:11.623+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:11.630 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:11.628+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:11.757 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:11.755+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:11.757 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:11.755+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:11.786 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:11.784+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:11.823 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:11.821+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:11.863 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:11.861+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:11.879 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:11.878+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:11.897 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:11.895+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:12.053 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:12.051+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:12.593 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:12.590+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:12.655 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:12.653+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:12.768 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:12.766+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:12.820 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:12.818+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:12.862 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:12.861+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:12.883 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:12.881+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:12.973 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:12.971+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:12.996 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:12.994+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:13.053 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:13.051+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:13.063 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:13.061+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:13.068 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:13.066+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:13.147 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:13.146+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:13.159 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:13.158+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:13.205 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:13.204+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:13.526 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:13.524+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:13.533 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:13.531+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:13.590 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:13.589+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:13.781 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:13.779+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:13.946 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:13.944+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:14.030 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:14.028+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:14.135 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:14.133+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:14.143 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:14.141+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:14.185 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:14.183+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:14.217 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:14.215+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:14.289 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:14.287+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:14.307 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:14.305+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:14.364 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:14.362+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:14.418 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:14.417+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:14.458 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:14.456+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:14.496 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:14.494+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:14.698 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:14.696+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:15.160 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.158+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:15.171 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.169+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:15.377 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.375+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:15.377 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.375+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:15.614 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.612+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:15.646 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.644+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:15.647 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.645+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:15.829 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.827+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:15.867 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.865+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:15.889 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.887+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:15.946 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.944+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:15.958 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.956+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:15.973 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.971+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:15.974 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.972+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:15.978 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.976+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:15.990 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:15.988+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:16.084 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:16.082+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:16.134 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:16.132+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:16.397 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:16.396+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:16.423 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:16.422+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:16.474 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:16.473+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:16.545 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:16.543+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:16.545 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:16.544+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:16.798 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:16.797+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:16.809 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:16.808+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:16.830 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:16.829+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:16.872 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:16.871+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:17.180 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:17.179+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:17.180 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:17.179+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:17.537 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:17.536+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:18.035 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.034+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:18.037 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.036+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:18.059 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.058+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:18.059 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.058+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:18.274 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.273+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:18.323 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.322+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:18.364 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.363+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:18.426 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.425+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:18.442 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.441+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:18.464 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.463+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:18.497 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.496+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:18.536 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.535+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:18.803 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.802+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:18.827 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.826+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:18.918 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.917+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:18.948 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.947+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a56c0 msgr2=0x7f1fe40809d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:18.960 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:18.959+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40845c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:19.271 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:19.270+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40809d0 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:19.606 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:19.605+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:19.809 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:19.808+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40809d0 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:19.904 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:19.903+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:19.968 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:19.967+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:20.027 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:20.026+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40809d0 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:20.091 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:20.090+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:20.186 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:20.185+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:20.287 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:20.286+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40809d0 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:20.303 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:20.302+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:20.347 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:20.346+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:20.669 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:20.668+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:20.681 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:20.680+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40809d0 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:20.778 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:20.777+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40809d0 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:20.995 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:20.994+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40809d0 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:21.073 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:21.072+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:21.084 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:21.083+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40809d0 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:21.113 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:21.112+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:21.123 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:21.122+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40809d0 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:21.377 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:21.376+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:21.392 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:21.391+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40809d0 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:21.417 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:21.416+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40809d0 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:21.465 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:21.464+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:21.480 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:21.479+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40809d0 msgr2=0x7f1fe40a56c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:21.517 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:21.516+0000 7f2005163640 0 -- 10.20.193.103:0/3609815496 >> v2:10.20.193.103:6804/3177587099 conn(0x7f1fe40a52f0 msgr2=0x7f1fe40a5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:21.566 INFO:teuthology.orchestra.run.trial103.stdout:Removed 27401 objects 2026-02-12T03:57:31.570 INFO:tasks.util.rados:rados -p unique_pool_0 bench 60 write --no-cleanup 2026-02-12T03:57:31.570 DEBUG:teuthology.orchestra.run.trial103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 bench 60 write --no-cleanup 2026-02-12T03:57:31.609 INFO:teuthology.orchestra.run.trial103.stdout:hints = 1 2026-02-12T03:57:31.611 INFO:teuthology.orchestra.run.trial103.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 60 seconds or 0 objects 2026-02-12T03:57:31.611 INFO:teuthology.orchestra.run.trial103.stdout:Object prefix: benchmark_data_trial103_33159 2026-02-12T03:57:31.629 INFO:teuthology.orchestra.run.trial103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-12T03:57:31.629 INFO:teuthology.orchestra.run.trial103.stdout: 0 0 0 0 0 0 - 0 2026-02-12T03:57:31.865 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:31.864+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x55c6dfc3c340 msgr2=0x7f79780811f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:31.912 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:31.911+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f7978080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:32.238 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:32.237+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x55c6dfc3c340 msgr2=0x7f7978083ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:32.239 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:32.237+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x55c6dfc3c340 msgr2=0x7f7978083ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:32.249 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:32.248+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f7978080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:32.442 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:32.440+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x55c6dfc3c340 msgr2=0x7f79780836e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:32.629 INFO:teuthology.orchestra.run.trial103.stdout: 1 16 386 370 1479.87 1480 0.0422871 0.0423227 2026-02-12T03:57:33.629 INFO:teuthology.orchestra.run.trial103.stdout: 2 16 643 627 1253.93 1028 0.0428308 0.0464914 2026-02-12T03:57:34.629 INFO:teuthology.orchestra.run.trial103.stdout: 3 16 935 919 1225.26 1168 0.235078 0.050451 2026-02-12T03:57:34.640 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:34.639+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f7978080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:34.962 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:34.961+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x55c6dfc3c340 msgr2=0x7f79780811f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:35.174 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:35.173+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x55c6dfc3c340 msgr2=0x7f79780811f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:35.387 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:35.387+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x55c6dfc3c340 msgr2=0x7f79780818b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:35.629 INFO:teuthology.orchestra.run.trial103.stdout: 4 16 1228 1212 1211.92 1172 0.0445441 0.0525131 2026-02-12T03:57:36.029 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:36.028+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f7978080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:36.030 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:36.029+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x55c6dfc3c340 msgr2=0x7f79780811f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:36.323 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:36.322+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x55c6dfc3c340 msgr2=0x7f79780811f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:36.629 INFO:teuthology.orchestra.run.trial103.stdout: 5 16 1526 1510 1207.92 1192 0.0425472 0.0527525 2026-02-12T03:57:36.715 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:36.714+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x55c6dfc3c340 msgr2=0x7f7978083b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:37.568 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:37.567+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780839a0 msgr2=0x7f7978083d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:37.629 INFO:teuthology.orchestra.run.trial103.stdout: 6 16 1827 1811 1207.25 1204 0.0561348 0.0527681 2026-02-12T03:57:37.807 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:37.806+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780839a0 msgr2=0x7f7978083d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:38.361 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:38.359+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f7978084320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:38.450 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:38.449+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f79780811f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:38.521 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:38.519+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780839a0 msgr2=0x7f7978083d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:38.630 INFO:teuthology.orchestra.run.trial103.stdout: 7 16 2198 2182 1246.77 1484 0.0495741 0.0511255 2026-02-12T03:57:38.699 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:38.698+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f7978084320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:39.192 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:39.191+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f79780842b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:39.456 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:39.455+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f79780842b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:39.630 INFO:teuthology.orchestra.run.trial103.stdout: 8 16 2523 2507 1253.42 1300 0.0435693 0.0500508 2026-02-12T03:57:40.360 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:40.358+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084b20 msgr2=0x7f7978083d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:40.375 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:40.374+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084b20 msgr2=0x7f7978083d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:40.601 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:40.600+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084b20 msgr2=0x7f7978083d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:40.617 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:40.616+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084b20 msgr2=0x7f7978083d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:40.629 INFO:teuthology.orchestra.run.trial103.stdout: 9 16 2792 2776 1233.7 1076 0.256806 0.0512749 2026-02-12T03:57:41.079 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:41.078+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f79780842b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:41.629 INFO:teuthology.orchestra.run.trial103.stdout: 10 16 3096 3080 1231.92 1216 0.0402488 0.051839 2026-02-12T03:57:41.645 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:41.644+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f79780842b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:41.733 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:41.732+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084b20 msgr2=0x7f7978083d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:41.781 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:41.780+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f79780842b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:42.052 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:42.051+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f79780842b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:42.630 INFO:teuthology.orchestra.run.trial103.stdout: 11 16 3234 3218 1170.11 552 0.0386434 0.0545999 2026-02-12T03:57:42.957 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:42.956+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084b20 msgr2=0x7f7978083d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:43.023 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:43.023+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f79780842b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:43.036 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:43.035+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084b20 msgr2=0x7f7978083d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:43.391 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:43.390+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f7978084140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:43.509 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:43.507+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978083d70 msgr2=0x7f79780811f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:43.630 INFO:teuthology.orchestra.run.trial103.stdout: 12 16 3539 3523 1174.26 1220 0.0466192 0.054393 2026-02-12T03:57:44.609 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:44.607+0000 7f798bfff640 0 -- 10.20.193.103:0/3727041652 >> [v2:10.20.193.103:3301/0,v1:10.20.193.103:6790/0] conn(0x55c6dfc4a3a0 msgr2=0x55c6dfc2b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:44.630 INFO:teuthology.orchestra.run.trial103.stdout: 13 16 3919 3903 1200.85 1520 0.0359271 0.0532142 2026-02-12T03:57:44.901 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:44.899+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f7978084140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:45.131 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:45.128+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f7978084140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:45.630 INFO:teuthology.orchestra.run.trial103.stdout: 14 16 4145 4129 1179.64 904 0.0470285 0.053428 2026-02-12T03:57:46.268 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:46.265+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f7978084140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:46.406 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:46.404+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f7978084140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:46.630 INFO:teuthology.orchestra.run.trial103.stdout: 15 16 4514 4498 1199.39 1476 0.0403992 0.053279 2026-02-12T03:57:46.870 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:46.868+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f7978084140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:47.411 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:47.409+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f7978084140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:47.459 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:47.457+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x55c6dfc4a3a0 msgr2=0x7f7978105b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:47.630 INFO:teuthology.orchestra.run.trial103.stdout: 16 16 4754 4738 1184.42 960 0.0353463 0.0533913 2026-02-12T03:57:48.630 INFO:teuthology.orchestra.run.trial103.stdout: 17 16 5126 5110 1202.27 1488 0.0396674 0.0531641 2026-02-12T03:57:49.630 INFO:teuthology.orchestra.run.trial103.stdout: 18 16 5516 5500 1222.14 1560 0.0362752 0.0523079 2026-02-12T03:57:50.338 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:50.336+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978060570 msgr2=0x7f7978084140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:50.630 INFO:teuthology.orchestra.run.trial103.stdout: 19 16 5816 5800 1220.97 1200 0.042787 0.0522932 2026-02-12T03:57:51.631 INFO:teuthology.orchestra.run.trial103.stdout:2026-02-12T03:57:51.630348+0000 min lat: 0.00660788 max lat: 0.654227 avg lat: 0.0528674 2026-02-12T03:57:51.631 INFO:teuthology.orchestra.run.trial103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-12T03:57:51.631 INFO:teuthology.orchestra.run.trial103.stdout: 20 16 6064 6048 1209.52 992 0.0389076 0.0528674 2026-02-12T03:57:52.126 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:52.124+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:52.422 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:52.420+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:52.493 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:52.491+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:52.630 INFO:teuthology.orchestra.run.trial103.stdout: 21 16 6360 6344 1208.3 1184 0.0418104 0.0529104 2026-02-12T03:57:53.058 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:53.056+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:53.489 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:53.487+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:53.631 INFO:teuthology.orchestra.run.trial103.stdout: 22 16 6620 6604 1200.65 1040 0.051573 0.0529606 2026-02-12T03:57:53.734 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:53.732+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:54.110 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:54.108+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:54.130 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:54.128+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:54.180 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:54.177+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:54.631 INFO:teuthology.orchestra.run.trial103.stdout: 23 16 6966 6950 1208.61 1384 0.0473241 0.0528951 2026-02-12T03:57:55.510 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:55.508+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:55.609 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:55.607+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:57:55.631 INFO:teuthology.orchestra.run.trial103.stdout: 24 16 7343 7327 1221.08 1508 0.0474129 0.0523604 2026-02-12T03:57:56.610 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:56.608+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:56.631 INFO:teuthology.orchestra.run.trial103.stdout: 25 16 7735 7719 1234.96 1568 0.0511611 0.0517663 2026-02-12T03:57:57.191 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:57.188+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:57.631 INFO:teuthology.orchestra.run.trial103.stdout: 26 16 8039 8023 1234.22 1216 0.0406765 0.0518161 2026-02-12T03:57:57.643 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:57.641+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:58.134 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:58.132+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:58.357 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:58.355+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:58.436 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:58.434+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:58.631 INFO:teuthology.orchestra.run.trial103.stdout: 27 16 8391 8375 1240.66 1408 0.0436205 0.0514222 2026-02-12T03:57:59.526 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:59.525+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:59.631 INFO:teuthology.orchestra.run.trial103.stdout: 28 16 8732 8716 1245.06 1364 0.0370633 0.0513661 2026-02-12T03:57:59.703 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:59.702+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:59.730 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:59.729+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:57:59.877 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:57:59.876+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:00.379 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:00.378+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:00.631 INFO:teuthology.orchestra.run.trial103.stdout: 29 16 9084 9068 1250.67 1408 0.0482754 0.0510603 2026-02-12T03:58:01.249 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:01.248+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:01.631 INFO:teuthology.orchestra.run.trial103.stdout: 30 16 9411 9395 1252.58 1308 0.0376785 0.0510579 2026-02-12T03:58:02.156 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:02.155+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:02.235 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:02.234+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:02.413 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:02.412+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:02.631 INFO:teuthology.orchestra.run.trial103.stdout: 31 16 9694 9678 1248.69 1132 0.241173 0.0510091 2026-02-12T03:58:03.631 INFO:teuthology.orchestra.run.trial103.stdout: 32 16 9989 9973 1246.54 1180 0.0427645 0.0513039 2026-02-12T03:58:03.652 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:03.650+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:03.891 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:03.889+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:04.632 INFO:teuthology.orchestra.run.trial103.stdout: 33 16 10220 10204 1236.76 924 0.0423457 0.0517101 2026-02-12T03:58:04.725 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:04.723+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:05.631 INFO:teuthology.orchestra.run.trial103.stdout: 34 16 10450 10434 1227.44 920 0.0371504 0.0521096 2026-02-12T03:58:06.632 INFO:teuthology.orchestra.run.trial103.stdout: 35 16 10835 10819 1236.37 1540 0.0445551 0.0517337 2026-02-12T03:58:07.279 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:07.277+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:58:07.509 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:07.508+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:07.632 INFO:teuthology.orchestra.run.trial103.stdout: 36 16 11124 11108 1234.14 1156 0.0382356 0.0518309 2026-02-12T03:58:08.142 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:08.141+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:08.142 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:08.141+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:08.273 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:08.272+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:08.476 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:08.475+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:58:08.632 INFO:teuthology.orchestra.run.trial103.stdout: 37 16 11490 11474 1240.35 1464 0.0359244 0.0515511 2026-02-12T03:58:08.845 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:08.844+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:08.904 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:08.903+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:09.181 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:09.180+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:09.632 INFO:teuthology.orchestra.run.trial103.stdout: 38 16 11640 11624 1223.49 600 0.0552112 0.0519829 2026-02-12T03:58:10.347 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:10.346+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:10.632 INFO:teuthology.orchestra.run.trial103.stdout: 39 16 11962 11946 1225.14 1288 0.0395692 0.0522101 2026-02-12T03:58:11.632 INFO:teuthology.orchestra.run.trial103.stdout:2026-02-12T03:58:11.631860+0000 min lat: 0.00575492 max lat: 0.654227 avg lat: 0.0522026 2026-02-12T03:58:11.632 INFO:teuthology.orchestra.run.trial103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-12T03:58:11.632 INFO:teuthology.orchestra.run.trial103.stdout: 40 16 12270 12254 1225.31 1232 0.0369858 0.0522026 2026-02-12T03:58:12.632 INFO:teuthology.orchestra.run.trial103.stdout: 41 16 12644 12628 1231.91 1496 0.0399325 0.0519236 2026-02-12T03:58:13.368 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:13.366+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:13.632 INFO:teuthology.orchestra.run.trial103.stdout: 42 16 13017 13001 1238.1 1492 0.0378743 0.0516609 2026-02-12T03:58:13.694 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:13.692+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:14.293 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:14.291+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:58:14.560 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:14.559+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:14.632 INFO:teuthology.orchestra.run.trial103.stdout: 43 16 13398 13382 1244.75 1524 0.0437033 0.0513859 2026-02-12T03:58:14.962 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:14.961+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:15.068 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:15.067+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:15.632 INFO:teuthology.orchestra.run.trial103.stdout: 44 16 13653 13637 1239.64 1020 0.0427614 0.0514662 2026-02-12T03:58:15.733 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:15.732+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:15.840 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:15.838+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:16.135 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:16.134+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:16.169 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:16.168+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:16.201 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:16.200+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:16.575 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:16.574+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:16.628 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:16.627+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:16.632 INFO:teuthology.orchestra.run.trial103.stdout: 45 16 13986 13970 1241.69 1332 0.0374847 0.0515137 2026-02-12T03:58:17.036 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:17.035+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:17.632 INFO:teuthology.orchestra.run.trial103.stdout: 46 16 14207 14191 1233.91 884 0.0427349 0.0518395 2026-02-12T03:58:17.898 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:17.896+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:58:17.939 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:17.938+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:18.241 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:18.240+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:58:18.633 INFO:teuthology.orchestra.run.trial103.stdout: 47 16 14565 14549 1238.12 1432 0.0509711 0.0516564 2026-02-12T03:58:19.369 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:19.367+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:19.633 INFO:teuthology.orchestra.run.trial103.stdout: 48 16 14946 14930 1244.08 1524 0.0484471 0.0514117 2026-02-12T03:58:20.181 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:20.180+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:20.633 INFO:teuthology.orchestra.run.trial103.stdout: 49 16 15251 15235 1243.58 1220 0.0285642 0.0514406 2026-02-12T03:58:20.814 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:20.813+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:21.371 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:21.369+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:21.397 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:21.396+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:21.633 INFO:teuthology.orchestra.run.trial103.stdout: 50 16 15614 15598 1247.75 1452 0.0458324 0.0512665 2026-02-12T03:58:21.682 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:21.681+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:58:21.872 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:21.871+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:22.426 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:22.425+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:58:22.633 INFO:teuthology.orchestra.run.trial103.stdout: 51 16 15833 15817 1240.46 876 0.0426117 0.0513613 2026-02-12T03:58:22.934 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:22.933+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:23.216 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:23.215+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:23.507 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:23.506+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:23.633 INFO:teuthology.orchestra.run.trial103.stdout: 52 16 16134 16118 1239.76 1204 0.0420618 0.0516008 2026-02-12T03:58:23.680 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:23.679+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:23.858 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:23.857+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:23.859 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:23.857+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:24.318 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:24.317+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:24.355 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:24.354+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:24.633 INFO:teuthology.orchestra.run.trial103.stdout: 53 16 16344 16328 1232.21 840 0.0491824 0.0519101 2026-02-12T03:58:25.633 INFO:teuthology.orchestra.run.trial103.stdout: 54 16 16616 16600 1229.54 1088 0.0437311 0.0519489 2026-02-12T03:58:25.964 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:25.962+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:25.964 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:25.963+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:26.345 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:26.344+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:26.499 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:26.498+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:26.500 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:26.499+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:26.633 INFO:teuthology.orchestra.run.trial103.stdout: 55 16 16835 16819 1223.11 876 0.039083 0.052177 2026-02-12T03:58:27.633 INFO:teuthology.orchestra.run.trial103.stdout: 56 16 17108 17092 1220.77 1092 0.0443235 0.0524009 2026-02-12T03:58:27.776 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:27.773+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:27.965 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:27.963+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:58:28.104 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:28.102+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:28.400 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:28.398+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:28.478 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:28.476+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:28.580 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:28.578+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:28.592 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:28.590+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:58:28.604 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:28.602+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:58:28.633 INFO:teuthology.orchestra.run.trial103.stdout: 57 16 17446 17430 1223.07 1352 0.0397385 0.0522622 2026-02-12T03:58:28.862 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:28.860+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:29.302 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:29.300+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:29.563 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:29.561+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:29.633 INFO:teuthology.orchestra.run.trial103.stdout: 58 16 17734 17718 1221.84 1152 0.0396251 0.0522956 2026-02-12T03:58:30.633 INFO:teuthology.orchestra.run.trial103.stdout: 59 16 17972 17956 1217.27 952 0.0506855 0.0525536 2026-02-12T03:58:31.136 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:31.133+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:58:31.277 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:31.275+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f7978084140 msgr2=0x7f7978106410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-12T03:58:31.528 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:58:31.526+0000 7f7990bc1640 0 -- 10.20.193.103:0/3727041652 >> v2:10.20.193.103:6804/3177587099 conn(0x7f79780e5af0 msgr2=0x7f7978105ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-12T03:58:31.633 INFO:teuthology.orchestra.run.trial103.stdout:2026-02-12T03:58:31.633305+0000 min lat: 0.00575492 max lat: 0.654227 avg lat: 0.0523814 2026-02-12T03:58:31.634 INFO:teuthology.orchestra.run.trial103.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-12T03:58:31.634 INFO:teuthology.orchestra.run.trial103.stdout: 60 14 18334 18320 1221.25 1456 0.0495736 0.0523814 2026-02-12T03:58:32.634 INFO:teuthology.orchestra.run.trial103.stdout:Total time run: 60.0306 2026-02-12T03:58:32.634 INFO:teuthology.orchestra.run.trial103.stdout:Total writes made: 18334 2026-02-12T03:58:32.634 INFO:teuthology.orchestra.run.trial103.stdout:Write size: 4194304 2026-02-12T03:58:32.634 INFO:teuthology.orchestra.run.trial103.stdout:Object size: 4194304 2026-02-12T03:58:32.634 INFO:teuthology.orchestra.run.trial103.stdout:Bandwidth (MB/sec): 1221.64 2026-02-12T03:58:32.634 INFO:teuthology.orchestra.run.trial103.stdout:Stddev Bandwidth: 243.079 2026-02-12T03:58:32.634 INFO:teuthology.orchestra.run.trial103.stdout:Max bandwidth (MB/sec): 1568 2026-02-12T03:58:32.634 INFO:teuthology.orchestra.run.trial103.stdout:Min bandwidth (MB/sec): 552 2026-02-12T03:58:32.634 INFO:teuthology.orchestra.run.trial103.stdout:Average IOPS: 305 2026-02-12T03:58:32.634 INFO:teuthology.orchestra.run.trial103.stdout:Stddev IOPS: 60.7698 2026-02-12T03:58:32.635 INFO:teuthology.orchestra.run.trial103.stdout:Max IOPS: 392 2026-02-12T03:58:32.635 INFO:teuthology.orchestra.run.trial103.stdout:Min IOPS: 138 2026-02-12T03:58:32.635 INFO:teuthology.orchestra.run.trial103.stdout:Average Latency(s): 0.0523744 2026-02-12T03:58:32.635 INFO:teuthology.orchestra.run.trial103.stdout:Stddev Latency(s): 0.0469848 2026-02-12T03:58:32.635 INFO:teuthology.orchestra.run.trial103.stdout:Max latency(s): 0.654227 2026-02-12T03:58:32.635 INFO:teuthology.orchestra.run.trial103.stdout:Min latency(s): 0.00575492 2026-02-12T03:58:32.640 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd df name osd.3 --format=json 2026-02-12T03:58:32.860 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:32.860 INFO:teuthology.orchestra.run.trial103.stdout:{"nodes":[{"id":3,"device_class":"ssd","name":"osd.3","type":"osd","type_id":0,"crush_weight":0.087890625,"depth":2,"pool_weights":{},"reweight":1,"kb":94371840,"kb_used":35214816,"kb_used_data":34574544,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":59157024,"utilization":37.314961751302086,"var":1,"pgs":1,"status":"up"}],"stray":[],"summary":{"total_kb":94371840,"total_kb_used":35214816,"total_kb_used_data":34574544,"total_kb_used_omap":14,"total_kb_used_meta":640241,"total_kb_avail":59157024,"average_utilization":37.314961751302086,"min_var":1,"max_var":1,"dev":0}} 2026-02-12T03:58:32.869 DEBUG:tasks.backfill_toofull:primary osd df: {'id': 3, 'device_class': 'ssd', 'name': 'osd.3', 'type': 'osd', 'type_id': 0, 'crush_weight': 0.087890625, 'depth': 2, 'pool_weights': {}, 'reweight': 1, 'kb': 94371840, 'kb_used': 35214816, 'kb_used_data': 34574544, 'kb_used_omap': 14, 'kb_used_meta': 640241, 'kb_avail': 59157024, 'utilization': 37.314961751302086, 'var': 1, 'pgs': 1, 'status': 'up'} 2026-02-12T03:58:32.869 INFO:tasks.backfill_toofull:test backfill reservation rejected with toofull 2026-02-12T03:58:32.870 DEBUG:tasks.backfill_toofull:update nearfull ratio to 0.3022511901855469 and backfillfull ratio to 0.3358346557617188 2026-02-12T03:58:32.870 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-nearfull-ratio 0.303 2026-02-12T03:58:33.175 INFO:teuthology.orchestra.run.trial103.stderr:osd set-nearfull-ratio 0.303 2026-02-12T03:58:33.189 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-backfillfull-ratio 0.337 2026-02-12T03:58:34.184 INFO:teuthology.orchestra.run.trial103.stderr:osd set-backfillfull-ratio 0.337 2026-02-12T03:58:34.196 DEBUG:tasks.backfill_toofull:start tartget osd 1 2026-02-12T03:58:34.196 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-12T03:58:34.197 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-12T03:58:34.199 INFO:tasks.ceph.osd.1:Started 2026-02-12T03:58:34.199 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-12T03:58:34.267 INFO:teuthology.orchestra.run.trial103.stderr:no valid command found; 10 closest matches: 2026-02-12T03:58:34.267 INFO:teuthology.orchestra.run.trial103.stderr:0 2026-02-12T03:58:34.267 INFO:teuthology.orchestra.run.trial103.stderr:1 2026-02-12T03:58:34.267 INFO:teuthology.orchestra.run.trial103.stderr:2 2026-02-12T03:58:34.267 INFO:teuthology.orchestra.run.trial103.stderr:abort 2026-02-12T03:58:34.268 INFO:teuthology.orchestra.run.trial103.stderr:assert 2026-02-12T03:58:34.268 INFO:teuthology.orchestra.run.trial103.stderr:bluestore collections 2026-02-12T03:58:34.268 INFO:teuthology.orchestra.run.trial103.stderr:bluestore compression stats [] 2026-02-12T03:58:34.268 INFO:teuthology.orchestra.run.trial103.stderr:bluestore list [] [] 2026-02-12T03:58:34.268 INFO:teuthology.orchestra.run.trial103.stderr:bluestore onode metadata 2026-02-12T03:58:34.268 INFO:teuthology.orchestra.run.trial103.stderr:config diff 2026-02-12T03:58:34.268 INFO:teuthology.orchestra.run.trial103.stderr:admin_socket: invalid command 2026-02-12T03:58:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-12T03:58:34.270 INFO:tasks.backfill_toofull.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-12T03:58:34.409 INFO:tasks.ceph.osd.1.trial103.stderr:2026-02-12T03:58:34.407+0000 7f1a73b97900 -1 Falling back to public interface 2026-02-12T03:58:39.271 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-12T03:58:39.341 INFO:teuthology.orchestra.run.trial103.stderr:no valid command found; 10 closest matches: 2026-02-12T03:58:39.341 INFO:teuthology.orchestra.run.trial103.stderr:0 2026-02-12T03:58:39.341 INFO:teuthology.orchestra.run.trial103.stderr:1 2026-02-12T03:58:39.341 INFO:teuthology.orchestra.run.trial103.stderr:2 2026-02-12T03:58:39.341 INFO:teuthology.orchestra.run.trial103.stderr:abort 2026-02-12T03:58:39.341 INFO:teuthology.orchestra.run.trial103.stderr:assert 2026-02-12T03:58:39.341 INFO:teuthology.orchestra.run.trial103.stderr:bluefs debug_inject_read_zeros 2026-02-12T03:58:39.342 INFO:teuthology.orchestra.run.trial103.stderr:bluefs files list 2026-02-12T03:58:39.342 INFO:teuthology.orchestra.run.trial103.stderr:bluefs stats 2026-02-12T03:58:39.342 INFO:teuthology.orchestra.run.trial103.stderr:bluestore allocator dump block 2026-02-12T03:58:39.342 INFO:teuthology.orchestra.run.trial103.stderr:bluestore allocator fragmentation block 2026-02-12T03:58:39.342 INFO:teuthology.orchestra.run.trial103.stderr:admin_socket: invalid command 2026-02-12T03:58:39.343 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-12T03:58:39.344 INFO:tasks.backfill_toofull.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-12T03:58:39.597 INFO:tasks.ceph.osd.1.trial103.stderr:2026-02-12T03:58:39.595+0000 7f1a73b97900 -1 osd.1 23 log_to_monitors true 2026-02-12T03:58:40.204 INFO:tasks.ceph.osd.1.trial103.stderr:2026-02-12T03:58:40.202+0000 7f1a67112640 -1 osd.1 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-12T03:58:44.345 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-12T03:58:44.419 INFO:teuthology.orchestra.run.trial103.stdout:{ 2026-02-12T03:58:44.419 INFO:teuthology.orchestra.run.trial103.stdout: "ops": [], 2026-02-12T03:58:44.419 INFO:teuthology.orchestra.run.trial103.stdout: "num_ops": 0 2026-02-12T03:58:44.419 INFO:teuthology.orchestra.run.trial103.stdout:} 2026-02-12T03:58:44.424 INFO:tasks.backfill_toofull.ceph_manager:waiting for peering to complete 2026-02-12T03:58:44.425 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:58:44.603 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:44.603 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:58:44.612 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2026-02-12T03:58:43.687712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":4,"acting":4,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564268,"kb_used_data":131214956,"kb_used_omap":56,"kb_used_meta":1349191,"kb_avail":244923092,"statfs":{"total":386547056640,"available":250801246208,"internally_reserved":0,"allocated":134364114944,"data_stored":134363456175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58094,"internal_metadata":1381571858},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000398"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73186,"reported_epoch":33,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:42.321686+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:42.321686+0000","last_peered":"2026-02-12T03:58:42.321686+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:42.321686+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":47,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2026-02-12T03:58:37.187825+0000","last_change":"2026-02-12T03:57:04.831581+0000","last_active":"2026-02-12T03:58:37.187825+0000","last_peered":"2026-02-12T03:58:37.187825+0000","last_clean":"2026-02-12T03:54:46.129360+0000","last_became_active":"2026-02-12T03:57:04.831581+0000","last_became_peered":"2026-02-12T03:57:04.831581+0000","last_unstale":"2026-02-12T03:58:37.187825+0000","last_undegraded":"2026-02-12T03:57:04.831331+0000","last_fullsized":"2026-02-12T03:57:04.831284+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T09:53:53.171130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705715,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188572,"kb_used_data":37548300,"kb_used_omap":18,"kb_used_meta":640237,"kb_avail":56183268,"statfs":{"total":96636764160,"available":57531666432,"internally_reserved":0,"allocated":38449459200,"data_stored":38449286150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19200,"internal_metadata":655602944},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188572,"kb_used_data":37548300,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183268,"statfs":{"total":96636764160,"available":57531666432,"internally_reserved":0,"allocated":38449459200,"data_stored":38449286150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159576,"kb_used_data":56117768,"kb_used_omap":4,"kb_used_meta":41787,"kb_avail":38212264,"statfs":{"total":96636764160,"available":39129358336,"internally_reserved":0,"allocated":57464594432,"data_stored":57464458397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4396,"internal_metadata":42790612},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705715,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27548,"kb_used_data":588,"kb_used_omap":18,"kb_used_meta":26925,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":602112,"data_stored":425478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19202,"internal_metadata":27571454},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:58:44.613 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:58:45.287 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:45.287 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:58:45.299 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2026-02-12T03:58:43.687712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":4,"acting":4,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564268,"kb_used_data":131214956,"kb_used_omap":56,"kb_used_meta":1349191,"kb_avail":244923092,"statfs":{"total":386547056640,"available":250801246208,"internally_reserved":0,"allocated":134364114944,"data_stored":134363456175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58094,"internal_metadata":1381571858},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000398"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73186,"reported_epoch":33,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:42.321686+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:42.321686+0000","last_peered":"2026-02-12T03:58:42.321686+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:42.321686+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":47,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2026-02-12T03:58:37.187825+0000","last_change":"2026-02-12T03:57:04.831581+0000","last_active":"2026-02-12T03:58:37.187825+0000","last_peered":"2026-02-12T03:58:37.187825+0000","last_clean":"2026-02-12T03:54:46.129360+0000","last_became_active":"2026-02-12T03:57:04.831581+0000","last_became_peered":"2026-02-12T03:57:04.831581+0000","last_unstale":"2026-02-12T03:58:37.187825+0000","last_undegraded":"2026-02-12T03:57:04.831331+0000","last_fullsized":"2026-02-12T03:57:04.831284+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T09:53:53.171130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705715,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188572,"kb_used_data":37548300,"kb_used_omap":18,"kb_used_meta":640237,"kb_avail":56183268,"statfs":{"total":96636764160,"available":57531666432,"internally_reserved":0,"allocated":38449459200,"data_stored":38449286150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19200,"internal_metadata":655602944},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188572,"kb_used_data":37548300,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183268,"statfs":{"total":96636764160,"available":57531666432,"internally_reserved":0,"allocated":38449459200,"data_stored":38449286150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159576,"kb_used_data":56117768,"kb_used_omap":4,"kb_used_meta":41787,"kb_avail":38212264,"statfs":{"total":96636764160,"available":39129358336,"internally_reserved":0,"allocated":57464594432,"data_stored":57464458397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4396,"internal_metadata":42790612},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705715,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27548,"kb_used_data":588,"kb_used_omap":18,"kb_used_meta":26925,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":602112,"data_stored":425478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19202,"internal_metadata":27571454},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:58:45.300 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:58:45.520 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:45.520 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 backfillfull osd(s)","count":2},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18337/55009 objects degraded (33.335%), 2 pgs degraded, 1 pg undersized","count":3},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":249,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745810432,"bytes_avail":250801246208,"bytes_total":386547056640,"degraded_objects":18337,"degraded_total":55009,"degraded_ratio":0.33334545256230796},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-12T03:58:43.687745+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:58:45.530 INFO:tasks.backfill_toofull.ceph_manager:{'fsid': 'e9633232-3262-4ad1-a60a-815554f29d9f', 'health': {'status': 'HEALTH_WARN', 'checks': {'OSDMAP_FLAGS': {'severity': 'HEALTH_WARN', 'summary': {'message': 'noout,nobackfill,norecover flag(s) set', 'count': 26}, 'muted': False}, 'OSD_BACKFILLFULL': {'severity': 'HEALTH_WARN', 'summary': {'message': '2 backfillfull osd(s)', 'count': 2}, 'muted': False}, 'PG_BACKFILL_FULL': {'severity': 'HEALTH_WARN', 'summary': {'message': "Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull", 'count': 1}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 18337/55009 objects degraded (33.335%), 2 pgs degraded, 1 pg undersized', 'count': 3}, 'muted': False}, 'POOL_BACKFILLFULL': {'severity': 'HEALTH_WARN', 'summary': {'message': '2 pool(s) backfillfull', 'count': 2}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 249, 'monmap': {'epoch': 1, 'min_mon_release_name': 'tentacle', 'num_mons': 3}, 'osdmap': {'epoch': 33, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1770868721, 'num_in_osds': 4, 'osd_in_since': 1770868478, 'num_remapped_pgs': 1}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+undersized+degraded', 'count': 1}, {'state_name': 'active+undersized+degraded+remapped+backfill_toofull', 'count': 1}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 18337, 'data_bytes': 76898697752, 'bytes_used': 135745810432, 'bytes_avail': 250801246208, 'bytes_total': 386547056640, 'degraded_objects': 18337, 'degraded_total': 55009, 'degraded_ratio': 0.33334545256230796}, 'fsmap': {'epoch': 1, 'btime': '2026-02-12T03:54:36:498858+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs'], 'services': {}}, 'servicemap': {'epoch': 5, 'modified': '2026-02-12T03:58:43.687745+0000', 'services': {'mon': {'daemons': {'summary': '', 'c': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'osd': {'daemons': {'summary': '', '1': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2026-02-12T03:58:45.530 INFO:tasks.backfill_toofull.ceph_manager:active! 2026-02-12T03:58:45.530 INFO:tasks.backfill_toofull.ceph_manager:waiting for osd.1 to be up 2026-02-12T03:58:45.531 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T03:58:45.710 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:45.711 INFO:teuthology.orchestra.run.trial103.stdout:{"epoch":33,"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","created":"2026-02-12T03:54:36.502189+0000","modified":"2026-02-12T03:58:42.307125+0000","last_up_change":"2026-02-12T03:58:41.199407+0000","last_in_change":"2026-02-12T03:54:38.267568+0000","flags":"noout,nobackfill,norecover,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801600,"flags_set":["nobackfill","noout","norecover","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.33700001239776611,"nearfull_ratio":0.30300000309944153,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T03:54:43.668614+0000","flags":4097,"flags_names":"hashpspool,backfillfull","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-12T03:54:50.692009+0000","flags":4097,"flags_names":"hashpspool,backfillfull","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_toofull","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"5abdfff6-6d88-4b2f-b2d5-4ef7df15e16f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6800","nonce":3386817029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6801","nonce":3386817029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6803","nonce":3386817029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6802","nonce":3386817029}]},"public_addr":"10.20.193.103:6800/3386817029","cluster_addr":"10.20.193.103:6801/3386817029","heartbeat_back_addr":"10.20.193.103:6803/3386817029","heartbeat_front_addr":"10.20.193.103:6802/3386817029","state":["exists","up"]},{"osd":1,"uuid":"619e1a4c-89fd-4b3f-89b2-59b67eff290c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":23,"up_from":31,"up_thru":31,"down_at":24,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6808","nonce":1320799238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6810","nonce":1320799238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6814","nonce":1320799238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6812","nonce":1320799238}]},"public_addr":"10.20.193.103:6808/1320799238","cluster_addr":"10.20.193.103:6810/1320799238","heartbeat_back_addr":"10.20.193.103:6814/1320799238","heartbeat_front_addr":"10.20.193.103:6812/1320799238","state":["exists","up"]},{"osd":2,"uuid":"d4795e0e-f032-4fa5-a993-f36b7e5b497f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6809","nonce":2960779923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6811","nonce":2960779923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6815","nonce":2960779923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6813","nonce":2960779923}]},"public_addr":"10.20.193.103:6809/2960779923","cluster_addr":"10.20.193.103:6811/2960779923","heartbeat_back_addr":"10.20.193.103:6815/2960779923","heartbeat_front_addr":"10.20.193.103:6813/2960779923","state":["backfillfull","exists","up"]},{"osd":3,"uuid":"4d822481-0e6c-42a6-b420-d6885a3f6cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6804","nonce":3177587099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6805","nonce":3177587099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6807","nonce":3177587099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.103:6806","nonce":3177587099}]},"public_addr":"10.20.193.103:6804/3177587099","cluster_addr":"10.20.193.103:6805/3177587099","heartbeat_back_addr":"10.20.193.103:6807/3177587099","heartbeat_front_addr":"10.20.193.103:6806/3177587099","state":["backfillfull","exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-12T03:54:40.249570+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-12T03:54:40.188543+0000","dead_epoch":24},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-12T03:54:40.188277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-12T03:54:40.185138+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"2.0","osds":[3,2147483647,2]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_toofull":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T03:58:45.720 INFO:tasks.backfill_toofull.ceph_manager:osd.1 is up 2026-02-12T03:58:45.720 DEBUG:tasks.backfill_toofull:waiting for pg 2.0 state is backfill_toofull 2026-02-12T03:58:50.721 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-12T03:58:50.722 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-12T03:58:50.722 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-12T03:58:50.722 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-12T03:58:50.814 INFO:teuthology.orchestra.run.trial103.stdout:38654705717 2026-02-12T03:58:50.815 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-12T03:58:50.816 INFO:teuthology.orchestra.run.trial103.stdout:133143986180 2026-02-12T03:58:50.816 INFO:teuthology.orchestra.run.trial103.stdout:38654705717 2026-02-12T03:58:50.817 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-12T03:58:50.818 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-12T03:58:50.819 INFO:teuthology.orchestra.run.trial103.stdout:38654705718 2026-02-12T03:58:50.820 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-12T03:58:51.040 INFO:teuthology.orchestra.run.trial103.stdout:38654705716 2026-02-12T03:58:51.041 INFO:teuthology.orchestra.run.trial103.stdout:38654705717 2026-02-12T03:58:51.044 INFO:teuthology.orchestra.run.trial103.stdout:38654705716 2026-02-12T03:58:51.044 INFO:teuthology.orchestra.run.trial103.stdout:133143986178 2026-02-12T03:58:51.052 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705717 got 38654705716 for osd.0 2026-02-12T03:58:51.053 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705718 got 38654705717 for osd.2 2026-02-12T03:58:51.054 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705717 got 38654705716 for osd.3 2026-02-12T03:58:51.056 INFO:tasks.backfill_toofull.ceph_manager:need seq 133143986180 got 133143986178 for osd.1 2026-02-12T03:58:52.053 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-12T03:58:52.054 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-12T03:58:52.055 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-12T03:58:52.056 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-12T03:58:52.240 INFO:teuthology.orchestra.run.trial103.stdout:38654705716 2026-02-12T03:58:52.241 INFO:teuthology.orchestra.run.trial103.stdout:38654705717 2026-02-12T03:58:52.252 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705718 got 38654705717 for osd.2 2026-02-12T03:58:52.253 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705717 got 38654705716 for osd.0 2026-02-12T03:58:52.280 INFO:teuthology.orchestra.run.trial103.stdout:133143986178 2026-02-12T03:58:52.280 INFO:teuthology.orchestra.run.trial103.stdout:38654705716 2026-02-12T03:58:52.291 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705717 got 38654705716 for osd.3 2026-02-12T03:58:52.292 INFO:tasks.backfill_toofull.ceph_manager:need seq 133143986180 got 133143986178 for osd.1 2026-02-12T03:58:53.253 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-12T03:58:53.254 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-12T03:58:53.292 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-12T03:58:53.292 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-12T03:58:53.443 INFO:teuthology.orchestra.run.trial103.stdout:38654705718 2026-02-12T03:58:53.444 INFO:teuthology.orchestra.run.trial103.stdout:38654705717 2026-02-12T03:58:53.455 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705718 got 38654705718 for osd.2 2026-02-12T03:58:53.455 DEBUG:teuthology.parallel:result is None 2026-02-12T03:58:53.456 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705717 got 38654705717 for osd.0 2026-02-12T03:58:53.456 DEBUG:teuthology.parallel:result is None 2026-02-12T03:58:53.478 INFO:teuthology.orchestra.run.trial103.stdout:38654705717 2026-02-12T03:58:53.480 INFO:teuthology.orchestra.run.trial103.stdout:133143986180 2026-02-12T03:58:53.488 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705717 got 38654705717 for osd.3 2026-02-12T03:58:53.489 DEBUG:teuthology.parallel:result is None 2026-02-12T03:58:53.491 INFO:tasks.backfill_toofull.ceph_manager:need seq 133143986180 got 133143986180 for osd.1 2026-02-12T03:58:53.491 DEBUG:teuthology.parallel:result is None 2026-02-12T03:58:53.491 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:58:53.710 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:53.710 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:58:53.719 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2026-02-12T03:58:53.688897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001184"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986180,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:58:53.720 INFO:tasks.backfill_toofull:pg={'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73189, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:58:46.364076+0000', 'last_change': '2026-02-12T03:58:42.321686+0000', 'last_active': '2026-02-12T03:58:46.364076+0000', 'last_peered': '2026-02-12T03:58:46.364076+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:58:46.364076+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:58:53.720 INFO:tasks.backfill_toofull:test pg not enter backfill_toofull after restarting backfill 2026-02-12T03:58:53.721 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df detail --format=json 2026-02-12T03:58:53.941 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:53.942 INFO:teuthology.orchestra.run.trial103.stdout:{"stats":{"total_bytes":386547056640,"total_avail_bytes":250801106944,"total_used_bytes":135745949696,"total_used_raw_bytes":135745949696,"total_used_raw_ratio":0.35117572546005249,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4},"stats_by_class":{"ssd":{"total_bytes":386547056640,"total_avail_bytes":250801106944,"total_used_bytes":135745949696,"total_used_raw_bytes":135745949696,"total_used_raw_ratio":0.35117572546005249}},"pools":[{"name":".mgr","id":1,"stats":{"stored":328192,"stored_data":328192,"stored_omap":0,"objects":2,"kb_used":648,"bytes_used":663552,"data_bytes_used":663552,"omap_bytes_used":0,"percent_used":4.8367242015956435e-06,"max_avail":68594855936,"quota_objects":0,"quota_bytes":0,"dirty":0,"rd":26,"rd_bytes":19456,"wr":37,"wr_bytes":329728,"compress_bytes_used":0,"compress_under_bytes":0,"stored_raw":656384,"avail_raw":137189715456}},{"name":"unique_pool_0","id":2,"stats":{"stored":89574965248,"stored_data":89574965248,"stored_omap":0,"objects":18335,"kb_used":131213324,"bytes_used":134362443776,"data_bytes_used":134362443776,"omap_bytes_used":0,"percent_used":0.49479427933692932,"max_avail":91459805184,"quota_objects":0,"quota_bytes":0,"dirty":0,"rd":1,"rd_bytes":1024,"wr":73139,"wr_bytes":191826495488,"compress_bytes_used":0,"compress_under_bytes":0,"stored_raw":134362447872,"avail_raw":137189715456}}]} 2026-02-12T03:58:53.951 DEBUG:tasks.backfill_toofull:pool unique_pool_0 df: {'stored': 89574965248, 'stored_data': 89574965248, 'stored_omap': 0, 'objects': 18335, 'kb_used': 131213324, 'bytes_used': 134362443776, 'data_bytes_used': 134362443776, 'omap_bytes_used': 0, 'percent_used': 0.4947942793369293, 'max_avail': 91459805184, 'quota_objects': 0, 'quota_bytes': 0, 'dirty': 0, 'rd': 1, 'rd_bytes': 1024, 'wr': 73139, 'wr_bytes': 191826495488, 'compress_bytes_used': 0, 'compress_under_bytes': 0, 'stored_raw': 134362447872, 'avail_raw': 137189715456} 2026-02-12T03:58:53.952 DEBUG:tasks.backfill_toofull:compress_ratio: 1.0 2026-02-12T03:58:53.952 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f66ecfb34c0370410e78b3ee641753d19da653b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_5f66ecfb34c0370410e78b3ee641753d19da653b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) ^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph-c_1279775fef08a9169052c6dd6a10581353958aac/qa/tasks/backfill_toofull.py", line 155, in task assert backfillfull < 0.9 ^^^^^^^^^^^^^^^^^^ AssertionError 2026-02-12T03:58:53.955 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-12T03:58:53.969 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-12T03:58:53.969 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:58:54.147 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:54.147 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:58:54.155 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2026-02-12T03:58:53.688897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001184"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986180,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:58:54.157 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:58:54.333 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:54.333 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:58:54.341 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2026-02-12T03:58:53.688897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001184"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986180,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:58:54.342 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:58:54.519 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:54.520 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:58:54.528 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2026-02-12T03:58:53.688897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001184"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986180,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:58:54.529 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:58:54.529 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73189, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:58:46.364076+0000', 'last_change': '2026-02-12T03:58:42.321686+0000', 'last_active': '2026-02-12T03:58:46.364076+0000', 'last_peered': '2026-02-12T03:58:46.364076+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:58:46.364076+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:58:54.529 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:58:54.748 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:54.749 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":258,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-12T03:58:43.687745+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:58:54.758 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:58:54.758 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:58:54.939 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:54.939 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:58:54.948 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2026-02-12T03:58:53.688897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001184"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986180,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:58:57.949 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:58:58.131 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:58.132 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:58:58.141 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2026-02-12T03:58:57.689430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.326189"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986181,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:58:58.142 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:58:58.320 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:58.320 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:58:58.329 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2026-02-12T03:58:57.689430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.326189"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986181,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:58:58.330 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:58:58.330 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73189, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:58:46.364076+0000', 'last_change': '2026-02-12T03:58:42.321686+0000', 'last_active': '2026-02-12T03:58:46.364076+0000', 'last_peered': '2026-02-12T03:58:46.364076+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:58:46.364076+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:58:58.330 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:58:58.553 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:58.553 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":262,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-12T03:58:43.687745+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:58:58.563 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:58:58.563 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:58:58.746 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:58:58.747 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:58:58.756 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2026-02-12T03:58:57.689430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.326189"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986181,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:01.758 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:01.936 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:01.936 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:01.945 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2026-02-12T03:59:01.689899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001467"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986182,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:01.946 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:02.125 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:02.126 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:02.135 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2026-02-12T03:59:01.689899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001467"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986182,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:02.136 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:02.136 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73189, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:58:46.364076+0000', 'last_change': '2026-02-12T03:58:42.321686+0000', 'last_active': '2026-02-12T03:58:46.364076+0000', 'last_peered': '2026-02-12T03:58:46.364076+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:58:46.364076+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:02.136 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:02.358 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:02.358 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":265,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-12T03:58:43.687745+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:02.368 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:02.368 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:02.551 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:02.551 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:02.561 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2026-02-12T03:59:01.689899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001467"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986182,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:05.562 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:05.742 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:05.742 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:05.751 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2026-02-12T03:59:05.690374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001477"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986183,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:05.752 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:05.931 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:05.932 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:05.940 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2026-02-12T03:59:05.690374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001477"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986183,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:05.941 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:05.941 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73189, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:58:46.364076+0000', 'last_change': '2026-02-12T03:58:42.321686+0000', 'last_active': '2026-02-12T03:58:46.364076+0000', 'last_peered': '2026-02-12T03:58:46.364076+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:58:46.364076+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:05.941 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:06.161 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:06.161 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":269,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-12T03:58:43.687745+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:06.171 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:06.172 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:06.347 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:06.348 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:06.356 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2026-02-12T03:59:05.690374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001477"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986183,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:09.357 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:09.538 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:09.538 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:09.548 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2026-02-12T03:59:07.690628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001481"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986183,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:09.549 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:09.728 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:09.729 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:09.737 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2026-02-12T03:59:09.690921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001491"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986183,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:09.738 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:09.738 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73189, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:58:46.364076+0000', 'last_change': '2026-02-12T03:58:42.321686+0000', 'last_active': '2026-02-12T03:58:46.364076+0000', 'last_peered': '2026-02-12T03:58:46.364076+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:58:46.364076+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:09.739 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:09.961 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:09.961 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":273,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-12T03:58:43.687745+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:09.970 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:09.970 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:10.150 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:10.150 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:10.161 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2026-02-12T03:59:09.690921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001491"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986183,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:13.162 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:13.344 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:13.344 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:13.353 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2026-02-12T03:59:11.691172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001543"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986184,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:13.354 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:13.530 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:13.531 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:13.539 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2026-02-12T03:59:11.691172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001543"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73189,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:58:46.364076+0000","last_change":"2026-02-12T03:58:42.321686+0000","last_active":"2026-02-12T03:58:46.364076+0000","last_peered":"2026-02-12T03:58:46.364076+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:58:46.364076+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986184,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:13.540 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:13.541 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73189, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:58:46.364076+0000', 'last_change': '2026-02-12T03:58:42.321686+0000', 'last_active': '2026-02-12T03:58:46.364076+0000', 'last_peered': '2026-02-12T03:58:46.364076+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:58:46.364076+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:13.541 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:13.763 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:13.764 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":277,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-12T03:58:43.687745+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:13.774 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:13.774 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:13.950 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:13.951 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:13.960 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2026-02-12T03:59:13.691459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001559"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986184,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:16.962 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:17.141 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:17.142 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:17.151 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2026-02-12T03:59:15.691713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001615"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986185,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:17.152 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:17.335 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:17.335 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:17.344 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2026-02-12T03:59:15.691713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001615"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986185,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:17.346 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:17.346 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73193, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:12.322415+0000', 'last_change': '2026-02-12T03:59:12.322415+0000', 'last_active': '2026-02-12T03:59:12.322415+0000', 'last_peered': '2026-02-12T03:59:12.322415+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:12.322415+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:17.346 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:17.570 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:17.570 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":281,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-12T03:58:43.687745+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:17.579 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:17.579 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:17.848 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T03:59:17.847+0000 7ff3fb7fe640 0 --2- 10.20.193.103:0/4089836792 >> [v2:10.20.193.103:3302/0,v1:10.20.193.103:6791/0] conn(0x7ff3fc11ab00 0x7ff3fc12aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T03:59:17.957 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:17.958 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:17.966 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2026-02-12T03:59:17.692011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001636"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986185,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:20.968 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:21.153 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:21.153 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:21.162 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2026-02-12T03:59:19.692264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001635"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986185,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:21.164 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:21.345 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:21.345 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:21.354 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2026-02-12T03:59:19.692264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001635"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986185,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:21.355 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:21.355 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73193, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:12.322415+0000', 'last_change': '2026-02-12T03:59:12.322415+0000', 'last_active': '2026-02-12T03:59:12.322415+0000', 'last_peered': '2026-02-12T03:59:12.322415+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:12.322415+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:21.355 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:21.574 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:21.574 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":285,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-12T03:58:43.687745+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:21.584 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:21.585 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:21.763 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:21.763 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:21.773 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2026-02-12T03:59:21.692555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001633"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986186,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:24.775 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:24.954 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:24.954 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:24.964 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2026-02-12T03:59:23.692794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986186,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:24.964 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:25.141 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:25.141 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:25.150 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2026-02-12T03:59:23.692794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986186,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:25.150 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:25.150 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73193, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:12.322415+0000', 'last_change': '2026-02-12T03:59:12.322415+0000', 'last_active': '2026-02-12T03:59:12.322415+0000', 'last_peered': '2026-02-12T03:59:12.322415+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:12.322415+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:25.150 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:25.368 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:25.368 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":288,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-12T03:58:43.687745+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:25.377 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:25.377 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:25.559 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:25.559 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:25.568 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2026-02-12T03:59:23.692794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986186,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:28.569 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:28.749 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:28.750 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:28.759 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2026-02-12T03:59:27.693374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001660"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986187,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:28.760 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:28.937 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:28.949 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:28.949 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2026-02-12T03:59:27.693374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001660"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986187,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:28.950 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:28.951 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73193, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:12.322415+0000', 'last_change': '2026-02-12T03:59:12.322415+0000', 'last_active': '2026-02-12T03:59:12.322415+0000', 'last_peered': '2026-02-12T03:59:12.322415+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:12.322415+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:28.951 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:29.176 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:29.176 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":292,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-12T03:58:43.687745+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:29.186 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:29.187 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:29.365 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:29.366 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:29.374 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2026-02-12T03:59:27.693374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001660"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986187,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:32.376 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:32.554 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:32.555 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:32.563 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2026-02-12T03:59:31.693933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001669"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986188,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:32.564 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:32.741 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:32.741 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:32.749 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2026-02-12T03:59:31.693933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001669"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986188,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:32.750 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:32.751 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73193, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:12.322415+0000', 'last_change': '2026-02-12T03:59:12.322415+0000', 'last_active': '2026-02-12T03:59:12.322415+0000', 'last_peered': '2026-02-12T03:59:12.322415+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:12.322415+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:32.751 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:32.987 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:32.987 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":296,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-12T03:58:43.687745+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:32.996 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:32.996 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:33.174 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:33.175 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:33.183 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2026-02-12T03:59:31.693933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001669"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986188,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:36.184 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:36.367 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:36.368 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:36.376 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2026-02-12T03:59:35.694388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986189,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:36.377 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:36.556 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:36.556 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:36.565 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2026-02-12T03:59:35.694388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986189,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:36.566 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:36.567 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73193, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:12.322415+0000', 'last_change': '2026-02-12T03:59:12.322415+0000', 'last_active': '2026-02-12T03:59:12.322415+0000', 'last_peered': '2026-02-12T03:59:12.322415+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:12.322415+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:36.567 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:36.792 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:36.792 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":300,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-12T03:58:43.687745+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:36.801 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:36.801 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:36.979 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:36.980 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:36.988 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2026-02-12T03:59:35.694388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986189,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:39.990 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:40.171 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:40.172 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:40.180 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2026-02-12T03:59:39.694863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001489"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986189,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:40.181 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:40.359 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:40.359 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:40.368 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2026-02-12T03:59:39.694863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001489"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986189,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:40.369 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:40.369 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73193, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:12.322415+0000', 'last_change': '2026-02-12T03:59:12.322415+0000', 'last_active': '2026-02-12T03:59:12.322415+0000', 'last_peered': '2026-02-12T03:59:12.322415+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:12.322415+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:40.370 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:40.599 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:40.600 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":304,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-12T03:59:37.694670+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:40.609 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:40.609 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:40.790 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:40.791 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:40.800 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2026-02-12T03:59:39.694863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001489"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73193,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:12.322415+0000","last_change":"2026-02-12T03:59:12.322415+0000","last_active":"2026-02-12T03:59:12.322415+0000","last_peered":"2026-02-12T03:59:12.322415+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:12.322415+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986189,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:43.802 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:43.978 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:43.978 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:43.987 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2026-02-12T03:59:43.695294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001360"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986191,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:43.988 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:44.170 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:44.170 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:44.180 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2026-02-12T03:59:43.695294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001360"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986191,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:44.181 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:44.181 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73197, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:42.323163+0000', 'last_change': '2026-02-12T03:59:42.323163+0000', 'last_active': '2026-02-12T03:59:42.323163+0000', 'last_peered': '2026-02-12T03:59:42.323163+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:42.323163+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:44.181 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:44.401 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:44.401 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":307,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-12T03:59:37.694670+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:44.410 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:44.411 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:44.591 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:44.592 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:44.600 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2026-02-12T03:59:43.695294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001360"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986191,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:47.601 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:47.780 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:47.780 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:47.789 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2026-02-12T03:59:47.695788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001400"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986192,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:47.789 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:47.966 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:47.966 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:47.974 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2026-02-12T03:59:47.695788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001400"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986192,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:47.975 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:47.975 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73197, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:42.323163+0000', 'last_change': '2026-02-12T03:59:42.323163+0000', 'last_active': '2026-02-12T03:59:42.323163+0000', 'last_peered': '2026-02-12T03:59:42.323163+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:42.323163+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:47.975 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:48.195 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:48.195 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":311,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-12T03:59:37.694670+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:48.204 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:48.204 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:48.383 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:48.384 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:48.392 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2026-02-12T03:59:47.695788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001400"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986192,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:51.394 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:51.574 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:51.575 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:51.584 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2026-02-12T03:59:49.696026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001393"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986192,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:51.585 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:51.767 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:51.767 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:51.776 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2026-02-12T03:59:51.696270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001407"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986193,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:51.778 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:51.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73197, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:42.323163+0000', 'last_change': '2026-02-12T03:59:42.323163+0000', 'last_active': '2026-02-12T03:59:42.323163+0000', 'last_peered': '2026-02-12T03:59:42.323163+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:42.323163+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:51.778 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:52.001 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:52.001 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":315,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-12T03:59:37.694670+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:52.010 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:52.011 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:52.191 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:52.191 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:52.200 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2026-02-12T03:59:51.696270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001407"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986193,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:55.202 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:55.380 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:55.380 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:55.389 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2026-02-12T03:59:53.696548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001469"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986193,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:55.390 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:55.566 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:55.567 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:55.575 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2026-02-12T03:59:53.696548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001469"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986193,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:55.576 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:55.576 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73197, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:42.323163+0000', 'last_change': '2026-02-12T03:59:42.323163+0000', 'last_active': '2026-02-12T03:59:42.323163+0000', 'last_peered': '2026-02-12T03:59:42.323163+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:42.323163+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:55.577 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:55.796 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:55.796 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":319,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-12T03:59:37.694670+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:55.805 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:55.806 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:55.985 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:55.986 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:55.994 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2026-02-12T03:59:55.696902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001608"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986194,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:58.996 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:59.175 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:59.176 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:59.184 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2026-02-12T03:59:57.697160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001676"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986194,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:59.185 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:59.364 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:59.365 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:59.373 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2026-02-12T03:59:57.697160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001676"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986194,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T03:59:59.374 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T03:59:59.374 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73197, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:42.323163+0000', 'last_change': '2026-02-12T03:59:42.323163+0000', 'last_active': '2026-02-12T03:59:42.323163+0000', 'last_peered': '2026-02-12T03:59:42.323163+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:42.323163+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T03:59:59.374 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T03:59:59.594 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:59.595 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":323,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-12T03:59:37.694670+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T03:59:59.604 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T03:59:59.604 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T03:59:59.780 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T03:59:59.781 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T03:59:59.790 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2026-02-12T03:59:59.697445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001656"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986194,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:02.792 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:02.977 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:02.978 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:02.987 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2026-02-12T04:00:01.697663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001637"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986195,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:02.988 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:03.169 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:03.169 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:03.178 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2026-02-12T04:00:01.697663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001637"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986195,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:03.178 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:03.178 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73197, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:42.323163+0000', 'last_change': '2026-02-12T03:59:42.323163+0000', 'last_active': '2026-02-12T03:59:42.323163+0000', 'last_peered': '2026-02-12T03:59:42.323163+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:42.323163+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:03.178 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:03.407 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:03.408 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":326,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-12T03:59:37.694670+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:03.417 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:03.417 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:03.592 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:03.593 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:03.601 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2026-02-12T04:00:01.697663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001637"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986195,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:06.603 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:06.781 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:06.782 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:06.790 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2026-02-12T04:00:05.698225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001676"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986196,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:06.791 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:06.971 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:06.972 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:06.982 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2026-02-12T04:00:05.698225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001676"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986196,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:06.983 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:06.983 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73197, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:42.323163+0000', 'last_change': '2026-02-12T03:59:42.323163+0000', 'last_active': '2026-02-12T03:59:42.323163+0000', 'last_peered': '2026-02-12T03:59:42.323163+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:42.323163+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:06.983 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:07.205 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:07.205 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":330,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-12T03:59:37.694670+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:07.215 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:07.215 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:07.396 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:07.397 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:07.405 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2026-02-12T04:00:05.698225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001676"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986196,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:10.406 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:10.586 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:10.587 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:10.596 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2026-02-12T04:00:09.698843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001683"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986196,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:10.597 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:10.777 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:10.777 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:10.786 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2026-02-12T04:00:09.698843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001683"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986196,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:10.787 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:10.787 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73197, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T03:59:42.323163+0000', 'last_change': '2026-02-12T03:59:42.323163+0000', 'last_active': '2026-02-12T03:59:42.323163+0000', 'last_peered': '2026-02-12T03:59:42.323163+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T03:59:42.323163+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:10.787 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:11.011 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:11.011 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":334,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:11.022 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:11.022 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:11.201 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:11.202 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:11.210 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2026-02-12T04:00:09.698843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001683"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73197,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T03:59:42.323163+0000","last_change":"2026-02-12T03:59:42.323163+0000","last_active":"2026-02-12T03:59:42.323163+0000","last_peered":"2026-02-12T03:59:42.323163+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T03:59:42.323163+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986196,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:14.211 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:14.391 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:14.391 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:14.400 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2026-02-12T04:00:13.699299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001635"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986198,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:14.401 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:14.579 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:14.580 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:14.589 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2026-02-12T04:00:13.699299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001635"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986198,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:14.589 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:14.589 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73201, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:12.324437+0000', 'last_change': '2026-02-12T04:00:12.324437+0000', 'last_active': '2026-02-12T04:00:12.324437+0000', 'last_peered': '2026-02-12T04:00:12.324437+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:12.324437+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:14.589 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:14.810 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:14.811 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":338,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:14.819 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:14.819 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:14.994 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:14.994 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:15.003 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2026-02-12T04:00:13.699299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001635"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986198,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:18.005 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:18.184 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:18.184 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:18.194 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2026-02-12T04:00:17.699892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001667"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986199,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:18.195 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:18.381 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:18.382 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:18.391 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2026-02-12T04:00:17.699892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001667"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986199,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:18.392 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:18.392 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73201, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:12.324437+0000', 'last_change': '2026-02-12T04:00:12.324437+0000', 'last_active': '2026-02-12T04:00:12.324437+0000', 'last_peered': '2026-02-12T04:00:12.324437+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:12.324437+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:18.392 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:18.611 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:18.611 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":342,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:18.620 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:18.620 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:18.798 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:18.798 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:18.807 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2026-02-12T04:00:17.699892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001667"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986199,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:21.807 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:21.985 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:21.985 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:21.994 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2026-02-12T04:00:21.700434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001590"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986200,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:21.996 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:22.179 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:22.180 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:22.188 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2026-02-12T04:00:21.700434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001590"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986200,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:22.189 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:22.190 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73201, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:12.324437+0000', 'last_change': '2026-02-12T04:00:12.324437+0000', 'last_active': '2026-02-12T04:00:12.324437+0000', 'last_peered': '2026-02-12T04:00:12.324437+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:12.324437+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:22.190 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:22.410 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:22.410 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":345,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:22.420 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:22.420 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:22.596 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:22.596 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:22.604 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2026-02-12T04:00:21.700434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001590"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986200,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:25.607 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:25.795 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:25.795 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:25.804 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2026-02-12T04:00:25.701007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001708"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986201,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:25.805 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:25.981 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:25.981 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:25.990 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2026-02-12T04:00:25.701007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001708"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986201,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:25.991 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:25.991 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73201, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:12.324437+0000', 'last_change': '2026-02-12T04:00:12.324437+0000', 'last_active': '2026-02-12T04:00:12.324437+0000', 'last_peered': '2026-02-12T04:00:12.324437+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:12.324437+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:25.992 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:26.213 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:26.213 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":349,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:26.222 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:26.223 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:26.402 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:26.403 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:26.412 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2026-02-12T04:00:25.701007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001708"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986201,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:29.414 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:29.597 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:29.597 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:29.606 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2026-02-12T04:00:27.701287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001705"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986201,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:29.607 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:29.785 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:29.785 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:29.794 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2026-02-12T04:00:29.701618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001725"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986201,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:29.795 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:29.795 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73201, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:12.324437+0000', 'last_change': '2026-02-12T04:00:12.324437+0000', 'last_active': '2026-02-12T04:00:12.324437+0000', 'last_peered': '2026-02-12T04:00:12.324437+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:12.324437+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:29.796 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:30.016 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:30.016 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":353,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:30.026 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:30.027 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:30.205 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:30.206 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:30.215 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2026-02-12T04:00:29.701618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001725"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986201,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:33.217 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:33.401 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:33.401 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:33.410 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2026-02-12T04:00:31.701893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001774"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986202,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:33.411 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:33.587 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:33.588 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:33.596 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2026-02-12T04:00:31.701893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001774"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986202,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:33.597 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:33.597 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73201, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:12.324437+0000', 'last_change': '2026-02-12T04:00:12.324437+0000', 'last_active': '2026-02-12T04:00:12.324437+0000', 'last_peered': '2026-02-12T04:00:12.324437+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:12.324437+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:33.598 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:33.821 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:33.821 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":357,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:33.830 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:33.831 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:34.009 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:34.009 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:34.019 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2026-02-12T04:00:33.702221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001786"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986202,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:37.020 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:37.199 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:37.199 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:37.208 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2026-02-12T04:00:35.702496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001831"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986203,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:37.209 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:37.387 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:37.388 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:37.397 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2026-02-12T04:00:35.702496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001831"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986203,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:37.398 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:37.398 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73201, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:12.324437+0000', 'last_change': '2026-02-12T04:00:12.324437+0000', 'last_active': '2026-02-12T04:00:12.324437+0000', 'last_peered': '2026-02-12T04:00:12.324437+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:12.324437+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:37.398 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:37.622 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:37.622 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":361,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:37.632 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:37.633 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:37.814 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:37.815 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:37.823 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2026-02-12T04:00:37.702806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001798"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986203,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:40.825 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:41.009 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:41.010 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:41.019 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2026-02-12T04:00:39.703007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001720"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986203,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:41.020 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:41.201 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:41.202 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:41.210 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2026-02-12T04:00:39.703007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001720"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986203,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:41.211 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:41.211 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73201, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:12.324437+0000', 'last_change': '2026-02-12T04:00:12.324437+0000', 'last_active': '2026-02-12T04:00:12.324437+0000', 'last_peered': '2026-02-12T04:00:12.324437+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:12.324437+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:41.212 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:41.438 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:41.439 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":364,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:41.448 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:41.449 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:41.627 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:41.627 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:41.636 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2026-02-12T04:00:39.703007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001720"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73201,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:12.324437+0000","last_change":"2026-02-12T04:00:12.324437+0000","last_active":"2026-02-12T04:00:12.324437+0000","last_peered":"2026-02-12T04:00:12.324437+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:12.324437+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364153+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364153+0000","last_peered":"2026-02-12T03:58:46.364153+0000","last_clean":"2026-02-12T03:58:46.364153+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364153+0000","last_undegraded":"2026-02-12T03:58:46.364153+0000","last_fullsized":"2026-02-12T03:58:46.364153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986203,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:44.638 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:44.815 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:44.815 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:44.824 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2026-02-12T04:00:43.703515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73205,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.324976+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.324976+0000","last_peered":"2026-02-12T04:00:42.324976+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.324976+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986204,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:44.824 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:45.002 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:45.003 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:45.012 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2026-02-12T04:00:43.703515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73205,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.324976+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.324976+0000","last_peered":"2026-02-12T04:00:42.324976+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.324976+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986204,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:45.013 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:45.013 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73205, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:42.324976+0000', 'last_change': '2026-02-12T04:00:42.324976+0000', 'last_active': '2026-02-12T04:00:42.324976+0000', 'last_peered': '2026-02-12T04:00:42.324976+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:42.324976+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:45.014 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:45.236 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:45.236 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":368,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:45.246 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:45.247 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:45.424 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:45.425 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:45.433 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2026-02-12T04:00:43.703515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73205,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.324976+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.324976+0000","last_peered":"2026-02-12T04:00:42.324976+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.324976+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986204,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:48.435 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:48.616 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:48.616 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:48.626 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2026-02-12T04:00:47.703982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001485"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73205,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.324976+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.324976+0000","last_peered":"2026-02-12T04:00:42.324976+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.324976+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986205,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:48.627 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:48.802 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:48.803 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:48.811 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2026-02-12T04:00:47.703982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001485"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73205,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.324976+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.324976+0000","last_peered":"2026-02-12T04:00:42.324976+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.324976+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986205,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:48.812 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:48.812 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73205, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:42.324976+0000', 'last_change': '2026-02-12T04:00:42.324976+0000', 'last_active': '2026-02-12T04:00:42.324976+0000', 'last_peered': '2026-02-12T04:00:42.324976+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:42.324976+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:48.813 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:49.042 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:49.043 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":372,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:49.052 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:49.052 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:49.229 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:49.229 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:49.237 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2026-02-12T04:00:47.703982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001485"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73205,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.324976+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.324976+0000","last_peered":"2026-02-12T04:00:42.324976+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.324976+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986205,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:52.239 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:52.421 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:52.422 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:52.432 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2026-02-12T04:00:51.704562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001555"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73205,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.324976+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.324976+0000","last_peered":"2026-02-12T04:00:42.324976+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.324976+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986206,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:52.433 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:52.609 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:52.610 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:52.618 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2026-02-12T04:00:51.704562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001555"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73205,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.324976+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.324976+0000","last_peered":"2026-02-12T04:00:42.324976+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.324976+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986206,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:52.619 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:52.619 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73205, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:42.324976+0000', 'last_change': '2026-02-12T04:00:42.324976+0000', 'last_active': '2026-02-12T04:00:42.324976+0000', 'last_peered': '2026-02-12T04:00:42.324976+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:42.324976+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:52.619 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:52.840 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:52.840 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":376,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:52.849 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:52.850 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:53.031 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:53.032 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:53.041 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2026-02-12T04:00:51.704562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001555"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73205,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.324976+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.324976+0000","last_peered":"2026-02-12T04:00:42.324976+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.324976+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986206,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:56.044 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:56.221 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:56.222 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:56.230 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2026-02-12T04:00:55.705198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001682"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986207,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:56.231 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:56.409 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:56.410 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:56.418 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2026-02-12T04:00:55.705198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001682"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986207,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:56.419 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:00:56.420 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73206, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:42.325007+0000', 'last_change': '2026-02-12T04:00:42.324976+0000', 'last_active': '2026-02-12T04:00:42.325007+0000', 'last_peered': '2026-02-12T04:00:42.325007+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:42.325007+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:00:56.420 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:00:56.640 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:56.640 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":380,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:00:56.650 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:00:56.650 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:00:56.826 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:00:56.826 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:00:56.835 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2026-02-12T04:00:55.705198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001682"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986207,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:00:59.837 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:00.015 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:00.015 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:00.024 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2026-02-12T04:00:59.705774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001791"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986207,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:00.026 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:00.204 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:00.204 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:00.213 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2026-02-12T04:00:59.705774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001791"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986207,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:00.214 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:00.214 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73206, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:42.325007+0000', 'last_change': '2026-02-12T04:00:42.324976+0000', 'last_active': '2026-02-12T04:00:42.325007+0000', 'last_peered': '2026-02-12T04:00:42.325007+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:42.325007+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:00.215 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:00.434 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:00.434 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":383,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:01:00.443 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:00.443 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:00.622 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:00.622 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:00.631 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2026-02-12T04:00:59.705774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001791"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986207,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:03.633 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:03.817 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:03.817 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:03.827 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2026-02-12T04:01:03.706226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001663"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986208,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:03.828 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:04.005 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:04.005 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:04.015 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2026-02-12T04:01:03.706226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001663"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986208,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:04.016 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:04.016 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73206, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:42.325007+0000', 'last_change': '2026-02-12T04:00:42.324976+0000', 'last_active': '2026-02-12T04:00:42.325007+0000', 'last_peered': '2026-02-12T04:00:42.325007+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:42.325007+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:04.017 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:04.236 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:04.236 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":387,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:01:04.245 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:04.246 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:04.424 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:04.424 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:04.433 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2026-02-12T04:01:03.706226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001663"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986208,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:07.434 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:07.615 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:07.615 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:07.624 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2026-02-12T04:01:05.706453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001617"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986209,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:07.625 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:07.803 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:07.803 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:07.812 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2026-02-12T04:01:07.706752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001553"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986209,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:07.813 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:07.813 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73206, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:42.325007+0000', 'last_change': '2026-02-12T04:00:42.324976+0000', 'last_active': '2026-02-12T04:00:42.325007+0000', 'last_peered': '2026-02-12T04:00:42.325007+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:42.325007+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:07.814 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:08.036 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:08.036 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":391,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:01:08.046 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:08.046 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:08.227 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:08.227 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:08.235 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2026-02-12T04:01:07.706752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001553"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986209,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:11.236 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:11.415 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:11.416 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:11.425 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2026-02-12T04:01:09.706984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001514"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986209,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:11.426 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:11.605 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:11.605 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:11.614 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2026-02-12T04:01:09.706984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001514"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986209,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:11.615 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:11.615 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73206, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:00:42.325007+0000', 'last_change': '2026-02-12T04:00:42.324976+0000', 'last_active': '2026-02-12T04:00:42.325007+0000', 'last_peered': '2026-02-12T04:00:42.325007+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:00:42.325007+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:11.616 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:11.839 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:11.839 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":395,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-12T04:00:09.698867+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:01:11.849 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:11.849 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:12.030 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:12.031 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:12.040 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2026-02-12T04:01:11.707282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001508"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73206,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:00:42.325007+0000","last_change":"2026-02-12T04:00:42.324976+0000","last_active":"2026-02-12T04:00:42.325007+0000","last_peered":"2026-02-12T04:00:42.325007+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:00:42.325007+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986210,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:15.042 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:15.219 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:15.220 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:15.229 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2026-02-12T04:01:13.707516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001526"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986210,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:15.229 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:15.407 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:15.408 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:15.416 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2026-02-12T04:01:13.707516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001526"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986210,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:15.417 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:15.417 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73210, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:12.325565+0000', 'last_change': '2026-02-12T04:01:12.325565+0000', 'last_active': '2026-02-12T04:01:12.325565+0000', 'last_peered': '2026-02-12T04:01:12.325565+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:12.325565+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:15.417 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:15.637 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:15.638 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":399,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:01:15.646 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:15.647 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:15.821 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:15.822 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:15.830 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2026-02-12T04:01:15.707768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001542"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986211,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:18.832 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:19.015 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:19.015 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:19.024 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2026-02-12T04:01:17.708009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001556"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986211,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:19.025 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:19.203 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:19.203 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:19.212 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2026-02-12T04:01:17.708009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001556"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986211,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:19.213 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:19.213 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73210, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:12.325565+0000', 'last_change': '2026-02-12T04:01:12.325565+0000', 'last_active': '2026-02-12T04:01:12.325565+0000', 'last_peered': '2026-02-12T04:01:12.325565+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:12.325565+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:19.213 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:19.434 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:19.435 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":402,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:01:19.444 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:19.445 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:19.624 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:19.625 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:19.633 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2026-02-12T04:01:17.708009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001556"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986211,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:22.635 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:22.817 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:22.818 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:22.826 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2026-02-12T04:01:21.708448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001464"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986212,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:22.828 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:23.006 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:23.006 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:23.015 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2026-02-12T04:01:21.708448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001464"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986212,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:23.016 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:23.016 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73210, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:12.325565+0000', 'last_change': '2026-02-12T04:01:12.325565+0000', 'last_active': '2026-02-12T04:01:12.325565+0000', 'last_peered': '2026-02-12T04:01:12.325565+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:12.325565+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:23.017 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:23.235 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:23.235 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":406,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:01:23.244 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:23.244 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:23.423 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:23.423 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:23.432 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2026-02-12T04:01:21.708448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001464"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986212,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:26.434 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:26.616 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:26.616 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:26.625 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2026-02-12T04:01:25.709051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001534"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986213,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:26.626 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:26.804 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:26.805 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:26.813 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2026-02-12T04:01:25.709051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001534"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986213,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:26.814 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:26.814 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73210, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:12.325565+0000', 'last_change': '2026-02-12T04:01:12.325565+0000', 'last_active': '2026-02-12T04:01:12.325565+0000', 'last_peered': '2026-02-12T04:01:12.325565+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:12.325565+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:26.815 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:27.031 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:27.032 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":410,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:01:27.041 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:27.042 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:27.218 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:27.218 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:27.226 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2026-02-12T04:01:25.709051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001534"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986213,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:30.228 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:30.407 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:30.407 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:30.416 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2026-02-12T04:01:29.709561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001551"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986213,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:30.417 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:30.596 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:30.597 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:30.606 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2026-02-12T04:01:29.709561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001551"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986213,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:30.607 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:30.607 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73210, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:12.325565+0000', 'last_change': '2026-02-12T04:01:12.325565+0000', 'last_active': '2026-02-12T04:01:12.325565+0000', 'last_peered': '2026-02-12T04:01:12.325565+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:12.325565+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:30.608 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:30.827 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:30.828 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":414,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:01:30.836 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:30.837 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:31.020 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:31.020 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:31.029 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2026-02-12T04:01:29.709561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001551"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986213,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:34.031 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:34.208 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:34.209 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:34.217 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2026-02-12T04:01:33.710106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001657"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986215,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:34.218 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:34.395 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:34.395 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:34.404 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2026-02-12T04:01:33.710106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001657"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986215,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:34.405 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:34.405 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73210, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:12.325565+0000', 'last_change': '2026-02-12T04:01:12.325565+0000', 'last_active': '2026-02-12T04:01:12.325565+0000', 'last_peered': '2026-02-12T04:01:12.325565+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:12.325565+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:34.405 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:34.628 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:34.628 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":418,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:01:34.638 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:34.638 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:34.817 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:34.818 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:34.827 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2026-02-12T04:01:33.710106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001657"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986215,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:37.829 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:38.010 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:38.010 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:38.019 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2026-02-12T04:01:37.710554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001502"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986216,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:38.020 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:38.197 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:38.198 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:38.206 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2026-02-12T04:01:37.710554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001502"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986216,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:38.207 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:38.207 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73210, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:12.325565+0000', 'last_change': '2026-02-12T04:01:12.325565+0000', 'last_active': '2026-02-12T04:01:12.325565+0000', 'last_peered': '2026-02-12T04:01:12.325565+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:12.325565+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:38.207 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:38.429 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:38.430 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":421,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:01:38.440 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:38.440 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:38.622 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:38.623 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:38.631 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2026-02-12T04:01:37.710554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001502"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986216,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:41.633 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:41.820 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:41.821 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:41.830 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2026-02-12T04:01:41.711091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986217,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:41.831 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:42.011 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:42.011 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:42.020 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2026-02-12T04:01:41.711091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986217,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:42.021 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:42.021 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73210, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:12.325565+0000', 'last_change': '2026-02-12T04:01:12.325565+0000', 'last_active': '2026-02-12T04:01:12.325565+0000', 'last_peered': '2026-02-12T04:01:12.325565+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:12.325565+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:42.022 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:42.252 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:42.252 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":425,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:01:42.262 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:42.262 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:42.444 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:42.444 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:42.454 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2026-02-12T04:01:41.711091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73210,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:12.325565+0000","last_change":"2026-02-12T04:01:12.325565+0000","last_active":"2026-02-12T04:01:12.325565+0000","last_peered":"2026-02-12T04:01:12.325565+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:12.325565+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986217,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:45.455 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:45.637 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:45.637 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:45.647 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2026-02-12T04:01:43.711317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001497"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986217,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:45.648 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:45.823 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:45.823 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:45.831 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2026-02-12T04:01:45.711637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001531"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986218,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:45.832 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:45.832 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73214, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:42.326119+0000', 'last_change': '2026-02-12T04:01:42.326119+0000', 'last_active': '2026-02-12T04:01:42.326119+0000', 'last_peered': '2026-02-12T04:01:42.326119+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:42.326119+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:45.833 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:46.056 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:46.057 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":429,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:01:46.067 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:46.067 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:46.245 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:46.245 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:46.254 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2026-02-12T04:01:45.711637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001531"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986218,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:49.256 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:49.434 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:49.434 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:49.443 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2026-02-12T04:01:47.711910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001566"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986218,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:49.445 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:49.625 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:49.625 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:49.634 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2026-02-12T04:01:47.711910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001566"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986218,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:49.634 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:49.634 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73214, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:42.326119+0000', 'last_change': '2026-02-12T04:01:42.326119+0000', 'last_active': '2026-02-12T04:01:42.326119+0000', 'last_peered': '2026-02-12T04:01:42.326119+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:42.326119+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:49.634 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:49.854 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:49.855 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":433,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:01:49.864 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:49.864 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:50.043 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:50.043 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:50.052 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2026-02-12T04:01:49.712253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001699"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986218,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:53.054 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:53.235 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:53.236 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:53.245 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2026-02-12T04:01:51.712511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001721"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986219,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:53.246 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:53.423 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:53.423 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:53.432 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2026-02-12T04:01:51.712511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001721"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986219,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:53.433 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:53.433 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73214, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:42.326119+0000', 'last_change': '2026-02-12T04:01:42.326119+0000', 'last_active': '2026-02-12T04:01:42.326119+0000', 'last_peered': '2026-02-12T04:01:42.326119+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:42.326119+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:53.433 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:53.654 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:53.654 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":437,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:01:53.664 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:53.665 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:53.843 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:53.844 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:53.852 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2026-02-12T04:01:53.712832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001741"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986219,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:56.855 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:57.034 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:57.034 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:57.043 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2026-02-12T04:01:55.713102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001785"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:57.044 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:57.221 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:57.222 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:57.230 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2026-02-12T04:01:55.713102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001785"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:01:57.231 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:01:57.231 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73214, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:42.326119+0000', 'last_change': '2026-02-12T04:01:42.326119+0000', 'last_active': '2026-02-12T04:01:42.326119+0000', 'last_peered': '2026-02-12T04:01:42.326119+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:42.326119+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:01:57.232 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:01:57.454 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:57.454 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":440,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:01:57.464 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:01:57.464 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:01:57.644 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:01:57.644 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:01:57.653 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2026-02-12T04:01:55.713102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001785"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:00.656 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:00.839 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:00.840 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:00.849 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2026-02-12T04:01:59.713612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:00.850 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:01.027 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:01.027 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:01.036 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2026-02-12T04:01:59.713612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:01.037 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:01.037 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73214, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:42.326119+0000', 'last_change': '2026-02-12T04:01:42.326119+0000', 'last_active': '2026-02-12T04:01:42.326119+0000', 'last_peered': '2026-02-12T04:01:42.326119+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:42.326119+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:01.038 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:01.257 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:01.257 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":444,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:02:01.267 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:01.267 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:01.447 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:01.447 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:01.455 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2026-02-12T04:01:59.713612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:04.457 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:04.636 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:04.636 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:04.644 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2026-02-12T04:02:03.714076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001564"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:04.646 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:04.823 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:04.823 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:04.832 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2026-02-12T04:02:03.714076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001564"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:04.833 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:04.833 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73214, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:42.326119+0000', 'last_change': '2026-02-12T04:01:42.326119+0000', 'last_active': '2026-02-12T04:01:42.326119+0000', 'last_peered': '2026-02-12T04:01:42.326119+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:42.326119+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:04.834 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:05.062 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:05.063 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":448,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:02:05.073 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:05.073 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:05.255 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:05.255 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:05.264 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2026-02-12T04:02:03.714076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001564"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:08.266 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:08.446 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:08.447 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:08.455 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2026-02-12T04:02:07.714528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001425"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:08.456 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:08.632 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:08.632 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:08.641 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2026-02-12T04:02:07.714528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001425"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:08.642 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:08.642 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73214, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:42.326119+0000', 'last_change': '2026-02-12T04:01:42.326119+0000', 'last_active': '2026-02-12T04:01:42.326119+0000', 'last_peered': '2026-02-12T04:01:42.326119+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:42.326119+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:08.642 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:08.868 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:08.869 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":452,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:02:08.877 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:08.877 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:09.052 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:09.052 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:09.061 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2026-02-12T04:02:07.714528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001425"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:12.062 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:12.243 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:12.243 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:12.252 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2026-02-12T04:02:11.715092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001480"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986223,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:12.253 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:12.431 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:12.431 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:12.440 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2026-02-12T04:02:11.715092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001480"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986223,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:12.440 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:12.440 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73214, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:01:42.326119+0000', 'last_change': '2026-02-12T04:01:42.326119+0000', 'last_active': '2026-02-12T04:01:42.326119+0000', 'last_peered': '2026-02-12T04:01:42.326119+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:01:42.326119+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:12.440 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:12.663 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:12.663 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":456,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:02:12.673 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:12.673 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:12.852 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:12.852 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:12.861 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2026-02-12T04:02:11.715092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001480"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73214,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:01:42.326119+0000","last_change":"2026-02-12T04:01:42.326119+0000","last_active":"2026-02-12T04:01:42.326119+0000","last_peered":"2026-02-12T04:01:42.326119+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:01:42.326119+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986223,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:15.863 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:16.041 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:16.042 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:16.050 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2026-02-12T04:02:15.715673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001597"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986224,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:16.051 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:16.235 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:16.235 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:16.243 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2026-02-12T04:02:15.715673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001597"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986224,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:16.244 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:16.245 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73218, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:12.326586+0000', 'last_change': '2026-02-12T04:02:12.326586+0000', 'last_active': '2026-02-12T04:02:12.326586+0000', 'last_peered': '2026-02-12T04:02:12.326586+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:12.326586+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:16.245 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:16.464 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:16.465 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":459,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:02:16.475 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:16.475 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:16.651 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:16.651 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:16.659 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2026-02-12T04:02:15.715673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001597"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986224,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:19.662 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:19.843 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:19.844 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:19.853 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2026-02-12T04:02:19.716147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001618"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986224,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:19.854 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:20.032 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:20.033 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:20.041 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2026-02-12T04:02:19.716147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001618"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986224,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:20.042 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:20.043 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73218, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:12.326586+0000', 'last_change': '2026-02-12T04:02:12.326586+0000', 'last_active': '2026-02-12T04:02:12.326586+0000', 'last_peered': '2026-02-12T04:02:12.326586+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:12.326586+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:20.043 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:20.266 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:20.267 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":463,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:02:20.276 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:20.277 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:20.454 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:20.455 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:20.464 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2026-02-12T04:02:19.716147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001618"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986224,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:23.466 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:23.647 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:23.648 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:23.657 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2026-02-12T04:02:21.716407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001612"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986225,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:23.658 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:23.838 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:23.839 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:23.847 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2026-02-12T04:02:23.716692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986225,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:23.848 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:23.848 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73218, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:12.326586+0000', 'last_change': '2026-02-12T04:02:12.326586+0000', 'last_active': '2026-02-12T04:02:12.326586+0000', 'last_peered': '2026-02-12T04:02:12.326586+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:12.326586+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:23.849 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:24.078 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:24.079 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":467,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:02:24.089 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:24.089 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:24.267 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:24.267 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:24.276 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2026-02-12T04:02:23.716692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986225,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:27.278 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:27.456 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:27.457 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:27.465 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2026-02-12T04:02:25.716985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001611"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986226,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:27.466 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:27.646 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:27.647 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:27.656 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2026-02-12T04:02:25.716985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001611"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986226,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:27.658 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:27.658 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73218, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:12.326586+0000', 'last_change': '2026-02-12T04:02:12.326586+0000', 'last_active': '2026-02-12T04:02:12.326586+0000', 'last_peered': '2026-02-12T04:02:12.326586+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:12.326586+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:27.659 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:27.878 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:27.878 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":471,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:02:27.888 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:27.888 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:28.061 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:28.062 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:28.070 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2026-02-12T04:02:27.717305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986226,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:31.072 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:31.251 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:31.252 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:31.260 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2026-02-12T04:02:29.717545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001631"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986226,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:31.261 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:31.442 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:31.442 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:31.450 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2026-02-12T04:02:29.717545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001631"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986226,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:31.451 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:31.452 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73218, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:12.326586+0000', 'last_change': '2026-02-12T04:02:12.326586+0000', 'last_active': '2026-02-12T04:02:12.326586+0000', 'last_peered': '2026-02-12T04:02:12.326586+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:12.326586+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:31.452 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:31.671 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:31.671 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":475,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-12T04:01:11.707309+0000","services":{}},"progress_events":{}} 2026-02-12T04:02:31.680 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:31.680 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:31.861 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:31.861 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:31.870 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2026-02-12T04:02:31.717798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001650"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986227,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:34.871 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:35.051 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:35.051 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:35.060 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2026-02-12T04:02:33.718035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001628"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986227,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:35.061 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:35.236 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:35.237 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:35.246 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2026-02-12T04:02:33.718035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001628"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986227,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:35.247 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:35.247 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73218, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:12.326586+0000', 'last_change': '2026-02-12T04:02:12.326586+0000', 'last_active': '2026-02-12T04:02:12.326586+0000', 'last_peered': '2026-02-12T04:02:12.326586+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:12.326586+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:35.247 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:35.518 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:02:35.518+0000 7f409b7fe640 0 --2- 10.20.193.103:0/3473412386 >> [v2:10.20.193.103:3302/0,v1:10.20.193.103:6791/0] conn(0x7f409c133f50 0x7f409c183f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:02:35.670 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:35.670 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":479,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":9,"modified":"2026-02-12T04:02:31.717825+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:02:35.680 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:35.680 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:35.857 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:35.858 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:35.866 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2026-02-12T04:02:35.718359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001667"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986228,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:38.868 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:39.048 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:39.048 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:39.057 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2026-02-12T04:02:37.718621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001635"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986228,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:39.059 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:39.236 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:39.237 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:39.246 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2026-02-12T04:02:37.718621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001635"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986228,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:39.247 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:39.247 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73218, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:12.326586+0000', 'last_change': '2026-02-12T04:02:12.326586+0000', 'last_active': '2026-02-12T04:02:12.326586+0000', 'last_peered': '2026-02-12T04:02:12.326586+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:12.326586+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:39.247 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:39.468 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:39.468 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":482,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":9,"modified":"2026-02-12T04:02:31.717825+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:02:39.478 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:39.478 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:39.657 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:39.657 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:39.666 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2026-02-12T04:02:37.718621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001635"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T03:58:46.364190+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T03:58:46.364190+0000","last_peered":"2026-02-12T03:58:46.364190+0000","last_clean":"2026-02-12T03:58:46.364190+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T03:58:46.364190+0000","last_undegraded":"2026-02-12T03:58:46.364190+0000","last_fullsized":"2026-02-12T03:58:46.364190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986228,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:42.668 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:42.851 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:42.851 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:42.860 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2026-02-12T04:02:41.719137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001592"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986229,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:42.861 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:43.037 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:43.038 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:43.046 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2026-02-12T04:02:41.719137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001592"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986229,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:43.047 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:43.048 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73218, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:12.326586+0000', 'last_change': '2026-02-12T04:02:12.326586+0000', 'last_active': '2026-02-12T04:02:12.326586+0000', 'last_peered': '2026-02-12T04:02:12.326586+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:12.326586+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:43.048 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:43.271 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:43.272 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":486,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":9,"modified":"2026-02-12T04:02:31.717825+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:02:43.282 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:43.282 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:43.461 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:43.462 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:43.471 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2026-02-12T04:02:41.719137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001592"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73218,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:12.326586+0000","last_change":"2026-02-12T04:02:12.326586+0000","last_active":"2026-02-12T04:02:12.326586+0000","last_peered":"2026-02-12T04:02:12.326586+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:12.326586+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986229,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:46.473 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:46.653 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:46.654 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:46.662 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2026-02-12T04:02:45.719717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001681"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73222,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327027+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327027+0000","last_peered":"2026-02-12T04:02:42.327027+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327027+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986230,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:46.664 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:46.843 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:46.844 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:46.852 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2026-02-12T04:02:45.719717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001681"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73222,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327027+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327027+0000","last_peered":"2026-02-12T04:02:42.327027+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327027+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986230,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:46.853 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:46.854 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73222, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:42.327027+0000', 'last_change': '2026-02-12T04:02:42.327027+0000', 'last_active': '2026-02-12T04:02:42.327027+0000', 'last_peered': '2026-02-12T04:02:42.327027+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:42.327027+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:46.854 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:47.074 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:47.074 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":490,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":9,"modified":"2026-02-12T04:02:31.717825+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:02:47.084 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:47.084 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:47.259 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:47.259 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:47.268 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2026-02-12T04:02:45.719717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001681"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73222,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327027+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327027+0000","last_peered":"2026-02-12T04:02:42.327027+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327027+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986230,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:50.269 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:50.460 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:50.460 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:50.469 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2026-02-12T04:02:49.720283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001662"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73222,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327027+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327027+0000","last_peered":"2026-02-12T04:02:42.327027+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327027+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986230,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:50.470 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:50.648 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:50.648 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:50.657 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2026-02-12T04:02:49.720283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001662"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73222,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327027+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327027+0000","last_peered":"2026-02-12T04:02:42.327027+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327027+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986230,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:50.658 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:50.658 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73222, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:42.327027+0000', 'last_change': '2026-02-12T04:02:42.327027+0000', 'last_active': '2026-02-12T04:02:42.327027+0000', 'last_peered': '2026-02-12T04:02:42.327027+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:42.327027+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:50.658 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:50.881 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:50.881 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":494,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":9,"modified":"2026-02-12T04:02:31.717825+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:02:50.891 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:50.891 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:51.162 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:51.162 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:51.174 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2026-02-12T04:02:49.720283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001662"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73222,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327027+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327027+0000","last_peered":"2026-02-12T04:02:42.327027+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327027+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986230,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:54.175 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:54.354 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:54.354 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:54.364 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2026-02-12T04:02:53.720798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001660"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986231,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:54.364 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:54.541 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:54.542 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:54.550 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2026-02-12T04:02:53.720798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001660"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986231,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:54.551 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:54.552 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73223, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:42.327061+0000', 'last_change': '2026-02-12T04:02:42.327027+0000', 'last_active': '2026-02-12T04:02:42.327061+0000', 'last_peered': '2026-02-12T04:02:42.327061+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:42.327061+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:54.552 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:54.773 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:54.773 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":498,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":9,"modified":"2026-02-12T04:02:31.717825+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:02:54.783 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:54.783 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:54.963 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:54.963 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:54.973 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2026-02-12T04:02:53.720798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001660"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986231,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:57.974 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:58.154 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:58.155 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:58.164 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2026-02-12T04:02:57.721320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986232,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:58.165 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:58.345 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:58.346 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:58.355 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2026-02-12T04:02:57.721320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986232,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:02:58.356 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:02:58.356 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73223, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:42.327061+0000', 'last_change': '2026-02-12T04:02:42.327027+0000', 'last_active': '2026-02-12T04:02:42.327061+0000', 'last_peered': '2026-02-12T04:02:42.327061+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:42.327061+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:02:58.356 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:02:58.575 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:58.575 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":502,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":9,"modified":"2026-02-12T04:02:31.717825+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:02:58.585 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:02:58.585 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:02:58.762 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:02:58.763 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:02:58.771 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2026-02-12T04:02:57.721320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986232,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:01.772 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:01.954 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:01.955 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:01.963 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2026-02-12T04:03:01.721752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001468"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986233,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:01.965 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:02.141 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:02.141 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:02.150 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2026-02-12T04:03:01.721752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001468"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986233,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:02.151 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:02.151 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73223, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:42.327061+0000', 'last_change': '2026-02-12T04:02:42.327027+0000', 'last_active': '2026-02-12T04:02:42.327061+0000', 'last_peered': '2026-02-12T04:02:42.327061+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:42.327061+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:02.151 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:02.372 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:02.373 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":505,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":9,"modified":"2026-02-12T04:02:31.717825+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:03:02.383 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:02.383 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:02.559 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:02.559 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:02.567 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2026-02-12T04:03:01.721752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001468"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986233,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:05.570 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:05.756 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:05.756 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:05.766 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2026-02-12T04:03:05.722293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001494"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986234,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:05.767 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:05.946 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:05.946 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:05.955 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2026-02-12T04:03:05.722293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001494"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986234,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:05.957 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:05.957 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73223, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:42.327061+0000', 'last_change': '2026-02-12T04:02:42.327027+0000', 'last_active': '2026-02-12T04:02:42.327061+0000', 'last_peered': '2026-02-12T04:02:42.327061+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:42.327061+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:05.957 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:06.179 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:06.179 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":509,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":9,"modified":"2026-02-12T04:02:31.717825+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:03:06.188 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:06.189 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:06.368 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:06.368 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:06.378 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2026-02-12T04:03:05.722293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001494"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986234,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:09.379 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:09.554 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:09.555 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:09.564 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2026-02-12T04:03:07.722547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001505"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986234,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:09.565 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:09.742 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:09.742 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:09.751 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2026-02-12T04:03:09.722809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001489"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986234,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:09.752 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:09.752 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73223, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:02:42.327061+0000', 'last_change': '2026-02-12T04:02:42.327027+0000', 'last_active': '2026-02-12T04:02:42.327061+0000', 'last_peered': '2026-02-12T04:02:42.327061+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:02:42.327061+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:09.753 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:09.973 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:09.973 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":513,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":9,"modified":"2026-02-12T04:02:31.717825+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:03:09.983 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:09.983 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:10.253 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:03:10.252+0000 7fb83d500640 0 --2- 10.20.193.103:0/2503859649 >> [v2:10.20.193.103:3302/0,v1:10.20.193.103:6791/0] conn(0x7fb838175760 0x7fb838195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:03:10.364 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:10.365 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:10.374 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2026-02-12T04:03:09.722809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001489"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986234,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:13.376 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:13.554 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:13.554 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:13.562 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2026-02-12T04:03:11.723056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001537"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73223,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:02:42.327061+0000","last_change":"2026-02-12T04:02:42.327027+0000","last_active":"2026-02-12T04:02:42.327061+0000","last_peered":"2026-02-12T04:02:42.327061+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:02:42.327061+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986235,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:13.564 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:13.742 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:13.742 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:13.751 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2026-02-12T04:03:13.723357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001605"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986235,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:13.752 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:13.752 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73227, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:12.327625+0000', 'last_change': '2026-02-12T04:03:12.327625+0000', 'last_active': '2026-02-12T04:03:12.327625+0000', 'last_peered': '2026-02-12T04:03:12.327625+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:12.327625+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:13.753 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:13.978 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:13.978 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":517,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":9,"modified":"2026-02-12T04:02:31.717825+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:03:13.987 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:13.988 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:14.168 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:14.169 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:14.177 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2026-02-12T04:03:13.723357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001605"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986235,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:17.179 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:17.359 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:17.360 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:17.368 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2026-02-12T04:03:15.723610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001607"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986236,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:17.369 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:17.547 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:17.548 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:17.557 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2026-02-12T04:03:15.723610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001607"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986236,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:17.558 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:17.558 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73227, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:12.327625+0000', 'last_change': '2026-02-12T04:03:12.327625+0000', 'last_active': '2026-02-12T04:03:12.327625+0000', 'last_peered': '2026-02-12T04:03:12.327625+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:12.327625+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:17.559 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:17.780 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:17.780 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":521,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":9,"modified":"2026-02-12T04:02:31.717825+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:03:17.790 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:17.790 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:18.862 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:18.862 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:18.872 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2026-02-12T04:03:17.723893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001600"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986236,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:21.874 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:22.063 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:22.063 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:22.071 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2026-02-12T04:03:21.724384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001574"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986237,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:22.073 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:22.253 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:22.253 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:22.263 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2026-02-12T04:03:21.724384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001574"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986237,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:22.264 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:22.264 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73227, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:12.327625+0000', 'last_change': '2026-02-12T04:03:12.327625+0000', 'last_active': '2026-02-12T04:03:12.327625+0000', 'last_peered': '2026-02-12T04:03:12.327625+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:12.327625+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:22.264 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:22.484 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:22.484 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":525,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:03:22.494 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:22.495 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:22.674 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:22.674 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:22.682 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2026-02-12T04:03:21.724384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001574"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986237,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:25.684 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:25.867 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:25.867 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:25.876 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2026-02-12T04:03:25.724965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001607"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986238,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:25.877 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:26.061 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:26.062 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:26.071 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2026-02-12T04:03:25.724965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001607"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986238,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:26.072 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:26.072 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73227, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:12.327625+0000', 'last_change': '2026-02-12T04:03:12.327625+0000', 'last_active': '2026-02-12T04:03:12.327625+0000', 'last_peered': '2026-02-12T04:03:12.327625+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:12.327625+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:26.072 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:26.292 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:26.292 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":529,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:03:26.302 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:26.302 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:26.480 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:26.481 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:26.490 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2026-02-12T04:03:25.724965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001607"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986238,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:29.492 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:29.672 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:29.672 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:29.681 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2026-02-12T04:03:27.725201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001591"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986238,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:29.681 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:29.862 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:29.863 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:29.872 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2026-02-12T04:03:29.725443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001549"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986238,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:29.873 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:29.873 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73227, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:12.327625+0000', 'last_change': '2026-02-12T04:03:12.327625+0000', 'last_active': '2026-02-12T04:03:12.327625+0000', 'last_peered': '2026-02-12T04:03:12.327625+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:12.327625+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:29.874 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:30.096 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:30.096 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":533,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:03:30.106 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:30.106 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:30.281 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:30.281 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:30.290 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2026-02-12T04:03:29.725443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001549"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986238,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:33.292 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:33.473 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:33.474 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:33.484 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2026-02-12T04:03:31.725714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001580"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986239,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:33.485 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:33.663 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:33.663 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:33.672 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2026-02-12T04:03:31.725714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001580"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986239,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:33.673 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:33.673 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73227, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:12.327625+0000', 'last_change': '2026-02-12T04:03:12.327625+0000', 'last_active': '2026-02-12T04:03:12.327625+0000', 'last_peered': '2026-02-12T04:03:12.327625+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:12.327625+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:33.673 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:33.893 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:33.894 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":537,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:03:33.903 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:33.903 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:34.081 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:34.082 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:34.091 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2026-02-12T04:03:33.726018+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001633"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986239,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:37.092 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:37.271 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:37.271 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:37.280 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2026-02-12T04:03:35.726266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001612"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986240,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:37.281 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:37.455 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:37.456 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:37.465 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2026-02-12T04:03:35.726266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001612"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986240,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:37.466 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:37.466 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73227, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:12.327625+0000', 'last_change': '2026-02-12T04:03:12.327625+0000', 'last_active': '2026-02-12T04:03:12.327625+0000', 'last_peered': '2026-02-12T04:03:12.327625+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:12.327625+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:37.467 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:37.697 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:37.697 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":541,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:03:37.706 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:37.707 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:37.885 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:37.885 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:37.894 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2026-02-12T04:03:37.726565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001600"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986240,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:40.896 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:41.076 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:41.076 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:41.085 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2026-02-12T04:03:39.726807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001605"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986240,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:41.087 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:41.264 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:41.265 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:41.273 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2026-02-12T04:03:39.726807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001605"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986240,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:41.274 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:41.274 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73227, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:12.327625+0000', 'last_change': '2026-02-12T04:03:12.327625+0000', 'last_active': '2026-02-12T04:03:12.327625+0000', 'last_peered': '2026-02-12T04:03:12.327625+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:12.327625+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:41.275 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:41.493 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:41.493 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":544,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:03:41.503 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:41.504 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:41.684 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:41.684 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:41.693 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2026-02-12T04:03:39.726807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001605"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73227,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:12.327625+0000","last_change":"2026-02-12T04:03:12.327625+0000","last_active":"2026-02-12T04:03:12.327625+0000","last_peered":"2026-02-12T04:03:12.327625+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:12.327625+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986240,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:44.696 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:44.876 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:44.876 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:44.885 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2026-02-12T04:03:43.727286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001571"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986241,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:44.886 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:45.063 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:45.063 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:45.072 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2026-02-12T04:03:43.727286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001571"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986241,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:45.073 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:45.073 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73231, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:42.328243+0000', 'last_change': '2026-02-12T04:03:42.328243+0000', 'last_active': '2026-02-12T04:03:42.328243+0000', 'last_peered': '2026-02-12T04:03:42.328243+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:42.328243+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:45.074 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:45.297 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:45.298 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":548,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:03:45.308 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:45.309 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:45.491 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:45.491 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:45.501 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2026-02-12T04:03:43.727286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001571"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986241,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:48.503 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:49.576 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:49.577 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:49.587 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2026-02-12T04:03:47.727737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001470"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986242,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:49.588 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:49.764 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:49.765 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:49.773 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2026-02-12T04:03:49.727997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001431"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986242,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:49.774 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:49.774 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73231, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:42.328243+0000', 'last_change': '2026-02-12T04:03:42.328243+0000', 'last_active': '2026-02-12T04:03:42.328243+0000', 'last_peered': '2026-02-12T04:03:42.328243+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:42.328243+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:49.775 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:49.992 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:49.993 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":553,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:03:50.002 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:50.002 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:50.180 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:50.180 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:50.189 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2026-02-12T04:03:49.727997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001431"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986242,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:53.191 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:53.373 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:53.373 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:53.382 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2026-02-12T04:03:51.728238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001431"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986243,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:53.383 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:53.565 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:53.565 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:53.573 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2026-02-12T04:03:51.728238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001431"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986243,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:53.574 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:53.574 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73231, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:42.328243+0000', 'last_change': '2026-02-12T04:03:42.328243+0000', 'last_active': '2026-02-12T04:03:42.328243+0000', 'last_peered': '2026-02-12T04:03:42.328243+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:42.328243+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:53.575 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:53.796 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:53.796 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":557,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:03:53.805 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:53.806 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:53.988 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:53.988 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:53.997 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2026-02-12T04:03:53.728460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001392"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986243,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:56.999 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:57.176 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:57.177 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:57.185 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2026-02-12T04:03:55.728688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001402"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986244,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:57.186 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:57.365 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:57.366 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:57.375 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2026-02-12T04:03:55.728688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001402"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986244,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:03:57.376 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:03:57.376 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73231, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:42.328243+0000', 'last_change': '2026-02-12T04:03:42.328243+0000', 'last_active': '2026-02-12T04:03:42.328243+0000', 'last_peered': '2026-02-12T04:03:42.328243+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:42.328243+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:03:57.376 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:03:57.596 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:57.596 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":561,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:03:57.606 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:03:57.606 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:03:57.782 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:03:57.783 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:03:57.792 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2026-02-12T04:03:57.728971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001447"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986244,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:00.794 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:00.974 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:00.974 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:00.982 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2026-02-12T04:03:59.729194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986244,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:00.984 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:01.161 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:01.161 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:01.170 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2026-02-12T04:03:59.729194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986244,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:01.171 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:01.171 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73231, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:42.328243+0000', 'last_change': '2026-02-12T04:03:42.328243+0000', 'last_active': '2026-02-12T04:03:42.328243+0000', 'last_peered': '2026-02-12T04:03:42.328243+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:42.328243+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:01.171 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:01.390 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:01.391 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":564,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:01.400 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:01.400 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:01.578 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:01.579 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:01.588 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2026-02-12T04:03:59.729194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986244,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:04.590 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:04.768 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:04.768 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:04.777 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2026-02-12T04:04:03.729652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001413"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:04.778 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:04.957 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:04.957 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:04.966 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2026-02-12T04:04:03.729652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001413"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:04.967 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:04.967 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73231, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:42.328243+0000', 'last_change': '2026-02-12T04:03:42.328243+0000', 'last_active': '2026-02-12T04:03:42.328243+0000', 'last_peered': '2026-02-12T04:03:42.328243+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:42.328243+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:04.968 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:05.189 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:05.189 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":568,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:05.199 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:05.199 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:05.376 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:05.377 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:05.385 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2026-02-12T04:04:03.729652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001413"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:08.387 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:08.567 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:08.568 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:08.577 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2026-02-12T04:04:07.730102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001413"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986246,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:08.578 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:08.758 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:08.758 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:08.767 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2026-02-12T04:04:07.730102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001413"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986246,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:08.768 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:08.768 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73231, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:42.328243+0000', 'last_change': '2026-02-12T04:03:42.328243+0000', 'last_active': '2026-02-12T04:03:42.328243+0000', 'last_peered': '2026-02-12T04:03:42.328243+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:42.328243+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:08.768 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:08.989 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:08.989 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":572,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:08.998 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:08.998 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:09.176 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:09.176 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:09.185 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2026-02-12T04:04:07.730102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001413"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986246,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:12.187 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:12.367 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:12.368 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:12.376 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2026-02-12T04:04:11.730528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001333"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:12.377 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:12.552 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:12.553 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:12.561 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2026-02-12T04:04:11.730528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001333"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:12.562 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:12.562 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73231, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:03:42.328243+0000', 'last_change': '2026-02-12T04:03:42.328243+0000', 'last_active': '2026-02-12T04:03:42.328243+0000', 'last_peered': '2026-02-12T04:03:42.328243+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:03:42.328243+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:12.563 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:12.792 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:12.793 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":576,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:12.802 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:12.803 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:12.983 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:12.983 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:12.992 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2026-02-12T04:04:11.730528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001333"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73231,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:03:42.328243+0000","last_change":"2026-02-12T04:03:42.328243+0000","last_active":"2026-02-12T04:03:42.328243+0000","last_peered":"2026-02-12T04:03:42.328243+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:03:42.328243+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:15.994 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:16.174 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:16.174 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:16.182 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2026-02-12T04:04:15.731033+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001381"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:16.184 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:16.359 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:16.360 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:16.368 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2026-02-12T04:04:15.731033+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001381"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:16.369 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:16.369 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73235, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:12.328800+0000', 'last_change': '2026-02-12T04:04:12.328800+0000', 'last_active': '2026-02-12T04:04:12.328800+0000', 'last_peered': '2026-02-12T04:04:12.328800+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:12.328800+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:16.370 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:16.590 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:16.590 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":580,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:16.600 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:16.600 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:16.776 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:16.777 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:16.785 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2026-02-12T04:04:15.731033+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001381"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:19.787 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:19.965 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:19.966 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:19.974 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2026-02-12T04:04:19.731554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001451"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:19.975 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:20.155 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:20.156 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:20.166 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2026-02-12T04:04:19.731554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001451"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:20.167 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:20.167 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73235, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:12.328800+0000', 'last_change': '2026-02-12T04:04:12.328800+0000', 'last_active': '2026-02-12T04:04:12.328800+0000', 'last_peered': '2026-02-12T04:04:12.328800+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:12.328800+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:20.167 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:20.388 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:20.388 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":583,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:20.397 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:20.398 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:20.576 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:20.577 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:20.585 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2026-02-12T04:04:19.731554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001451"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:23.587 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:23.767 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:23.768 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:23.776 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2026-02-12T04:04:23.732058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986249,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:23.777 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:23.955 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:23.955 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:23.963 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2026-02-12T04:04:23.732058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986249,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:23.964 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:23.965 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73235, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:12.328800+0000', 'last_change': '2026-02-12T04:04:12.328800+0000', 'last_active': '2026-02-12T04:04:12.328800+0000', 'last_peered': '2026-02-12T04:04:12.328800+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:12.328800+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:23.965 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:24.186 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:24.186 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":587,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:24.195 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:24.195 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:24.373 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:24.374 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:24.383 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2026-02-12T04:04:23.732058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986249,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:27.384 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:27.566 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:27.567 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:27.576 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2026-02-12T04:04:25.732345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001552"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986250,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:27.577 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:27.754 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:27.754 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:27.764 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2026-02-12T04:04:27.732677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001643"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986250,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:27.765 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:27.765 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73235, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:12.328800+0000', 'last_change': '2026-02-12T04:04:12.328800+0000', 'last_active': '2026-02-12T04:04:12.328800+0000', 'last_peered': '2026-02-12T04:04:12.328800+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:12.328800+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:27.766 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:27.983 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:27.984 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":591,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:27.993 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:27.993 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:29.066 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:29.066 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:29.078 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2026-02-12T04:04:27.732677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001643"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986250,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:32.080 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:32.257 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:32.258 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:32.267 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2026-02-12T04:04:31.733139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001584"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986251,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:32.268 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:32.443 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:32.443 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:32.451 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2026-02-12T04:04:31.733139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001584"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986251,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:32.452 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:32.453 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73235, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:12.328800+0000', 'last_change': '2026-02-12T04:04:12.328800+0000', 'last_active': '2026-02-12T04:04:12.328800+0000', 'last_peered': '2026-02-12T04:04:12.328800+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:12.328800+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:32.453 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:32.672 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:32.672 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":596,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:32.682 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:32.682 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:32.865 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:32.866 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:32.875 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2026-02-12T04:04:31.733139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001584"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986251,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:35.877 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:36.052 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:36.053 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:36.061 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2026-02-12T04:04:35.733665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001606"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986252,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:36.061 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:36.235 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:36.236 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:36.244 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2026-02-12T04:04:35.733665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001606"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986252,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:36.245 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:36.245 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73235, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:12.328800+0000', 'last_change': '2026-02-12T04:04:12.328800+0000', 'last_active': '2026-02-12T04:04:12.328800+0000', 'last_peered': '2026-02-12T04:04:12.328800+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:12.328800+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:36.246 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:36.462 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:36.463 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":599,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:36.473 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:36.473 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:36.650 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:36.651 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:36.659 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2026-02-12T04:04:35.733665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001606"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986252,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:39.661 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:39.840 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:39.840 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:39.849 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2026-02-12T04:04:39.734190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001512"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986252,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:39.850 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:40.027 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:40.028 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:40.036 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2026-02-12T04:04:39.734190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001512"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986252,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:40.037 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:40.037 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73235, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:12.328800+0000', 'last_change': '2026-02-12T04:04:12.328800+0000', 'last_active': '2026-02-12T04:04:12.328800+0000', 'last_peered': '2026-02-12T04:04:12.328800+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:12.328800+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:40.037 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:40.266 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:40.267 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":603,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:40.277 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:40.277 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:40.455 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:40.455 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:40.464 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2026-02-12T04:04:39.734190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001512"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:00:40.216144+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:00:40.216144+0000","last_peered":"2026-02-12T04:00:40.216144+0000","last_clean":"2026-02-12T04:00:40.216144+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:00:40.216144+0000","last_undegraded":"2026-02-12T04:00:40.216144+0000","last_fullsized":"2026-02-12T04:00:40.216144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986252,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:43.466 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:43.650 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:43.650 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:43.659 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2026-02-12T04:04:41.734432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001537"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73235,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:12.328800+0000","last_change":"2026-02-12T04:04:12.328800+0000","last_active":"2026-02-12T04:04:12.328800+0000","last_peered":"2026-02-12T04:04:12.328800+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:12.328800+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986253,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:43.660 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:43.837 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:43.838 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:43.847 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2026-02-12T04:04:43.734652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73239,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329280+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329280+0000","last_peered":"2026-02-12T04:04:42.329280+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329280+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986253,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:43.848 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:43.848 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73239, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:42.329280+0000', 'last_change': '2026-02-12T04:04:42.329280+0000', 'last_active': '2026-02-12T04:04:42.329280+0000', 'last_peered': '2026-02-12T04:04:42.329280+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:42.329280+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:43.849 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:44.068 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:44.068 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":607,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:44.077 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:44.078 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:44.259 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:44.260 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:44.269 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2026-02-12T04:04:43.734652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73239,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329280+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329280+0000","last_peered":"2026-02-12T04:04:42.329280+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329280+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986253,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:47.271 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:47.452 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:47.452 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:47.461 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2026-02-12T04:04:45.734887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001469"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73239,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329280+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329280+0000","last_peered":"2026-02-12T04:04:42.329280+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329280+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986254,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:47.462 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:47.641 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:47.642 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:47.650 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2026-02-12T04:04:45.734887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001469"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73239,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329280+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329280+0000","last_peered":"2026-02-12T04:04:42.329280+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329280+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986254,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:47.651 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:47.651 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73239, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:42.329280+0000', 'last_change': '2026-02-12T04:04:42.329280+0000', 'last_active': '2026-02-12T04:04:42.329280+0000', 'last_peered': '2026-02-12T04:04:42.329280+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:42.329280+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:47.652 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:47.873 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:47.873 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":611,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:47.884 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:47.884 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:48.062 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:48.063 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:48.071 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2026-02-12T04:04:47.735120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001454"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73239,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329280+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329280+0000","last_peered":"2026-02-12T04:04:42.329280+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329280+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986254,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:51.074 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:51.256 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:51.256 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:51.265 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2026-02-12T04:04:49.735357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001427"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73239,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329280+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329280+0000","last_peered":"2026-02-12T04:04:42.329280+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329280+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986254,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:51.266 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:51.441 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:51.442 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:51.450 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2026-02-12T04:04:49.735357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001427"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73239,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329280+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329280+0000","last_peered":"2026-02-12T04:04:42.329280+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329280+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986254,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:51.451 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:51.451 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73239, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:42.329280+0000', 'last_change': '2026-02-12T04:04:42.329280+0000', 'last_active': '2026-02-12T04:04:42.329280+0000', 'last_peered': '2026-02-12T04:04:42.329280+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:42.329280+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:51.452 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:51.674 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:51.675 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":615,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:51.685 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:51.685 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:51.861 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:51.861 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:51.871 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2026-02-12T04:04:51.735671+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001480"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73239,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329280+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329280+0000","last_peered":"2026-02-12T04:04:42.329280+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329280+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986255,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:54.872 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:55.051 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:55.052 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:55.061 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2026-02-12T04:04:53.735921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001488"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986255,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:55.062 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:55.240 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:55.241 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:55.250 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2026-02-12T04:04:53.735921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001488"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986255,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:55.251 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:55.251 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73240, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:42.329355+0000', 'last_change': '2026-02-12T04:04:42.329280+0000', 'last_active': '2026-02-12T04:04:42.329355+0000', 'last_peered': '2026-02-12T04:04:42.329355+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:42.329355+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:55.252 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:55.473 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:55.473 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":618,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:55.483 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:55.484 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:55.663 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:55.664 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:55.673 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2026-02-12T04:04:53.735921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001488"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986255,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:58.675 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:58.856 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:58.857 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:58.866 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2026-02-12T04:04:57.736425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001537"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986256,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:58.867 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:59.051 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:59.052 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:59.061 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2026-02-12T04:04:57.736425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001537"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986256,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:04:59.062 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:04:59.062 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73240, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:42.329355+0000', 'last_change': '2026-02-12T04:04:42.329280+0000', 'last_active': '2026-02-12T04:04:42.329355+0000', 'last_peered': '2026-02-12T04:04:42.329355+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:42.329355+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:04:59.062 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:04:59.282 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:59.283 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":622,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:04:59.292 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:04:59.292 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:04:59.468 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:04:59.469 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:04:59.477 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2026-02-12T04:04:57.736425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001537"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986256,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:02.479 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:02.660 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:02.661 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:02.670 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2026-02-12T04:05:01.736871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986257,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:02.671 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:02.853 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:02.853 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:02.863 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2026-02-12T04:05:01.736871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986257,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:02.864 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:02.864 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73240, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:42.329355+0000', 'last_change': '2026-02-12T04:04:42.329280+0000', 'last_active': '2026-02-12T04:04:42.329355+0000', 'last_peered': '2026-02-12T04:04:42.329355+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:42.329355+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:02.865 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:03.085 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:03.085 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":626,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:03.095 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:03.095 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:03.274 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:03.274 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:03.283 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2026-02-12T04:05:01.736871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986257,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:06.285 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:06.465 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:06.465 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:06.474 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2026-02-12T04:05:05.737394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001472"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986258,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:06.475 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:06.651 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:06.651 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:06.660 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2026-02-12T04:05:05.737394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001472"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986258,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:06.661 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:06.662 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73240, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:42.329355+0000', 'last_change': '2026-02-12T04:04:42.329280+0000', 'last_active': '2026-02-12T04:04:42.329355+0000', 'last_peered': '2026-02-12T04:04:42.329355+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:42.329355+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:06.662 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:06.880 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:06.880 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":630,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:06.890 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:06.890 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:07.075 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:07.076 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:07.085 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2026-02-12T04:05:05.737394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001472"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986258,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:10.087 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:10.270 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:10.270 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:10.279 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2026-02-12T04:05:09.737929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001504"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986258,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:10.280 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:10.456 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:10.456 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:10.465 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2026-02-12T04:05:09.737929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001504"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986258,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:10.466 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:10.466 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73240, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:04:42.329355+0000', 'last_change': '2026-02-12T04:04:42.329280+0000', 'last_active': '2026-02-12T04:04:42.329355+0000', 'last_peered': '2026-02-12T04:04:42.329355+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:04:42.329355+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:10.467 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:10.688 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:10.688 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":634,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:10.697 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:10.697 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:10.875 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:10.875 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:10.885 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2026-02-12T04:05:09.737929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001504"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73240,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:04:42.329355+0000","last_change":"2026-02-12T04:04:42.329280+0000","last_active":"2026-02-12T04:04:42.329355+0000","last_peered":"2026-02-12T04:04:42.329355+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:04:42.329355+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986258,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:13.886 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:14.063 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:14.064 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:14.073 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2026-02-12T04:05:13.738428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001557"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:14.074 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:14.250 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:14.251 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:14.259 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2026-02-12T04:05:13.738428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001557"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:14.260 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:14.260 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73244, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:12.330081+0000', 'last_change': '2026-02-12T04:05:12.330081+0000', 'last_active': '2026-02-12T04:05:12.330081+0000', 'last_peered': '2026-02-12T04:05:12.330081+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:12.330081+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:14.261 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:14.481 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:14.481 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":637,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:14.490 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:14.491 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:14.670 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:14.670 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:14.679 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2026-02-12T04:05:13.738428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001557"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:17.681 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:17.860 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:17.860 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:17.869 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2026-02-12T04:05:17.738979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001585"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:17.870 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:18.942 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:18.943 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:18.953 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2026-02-12T04:05:17.738979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001585"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:18.954 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:18.954 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73244, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:12.330081+0000', 'last_change': '2026-02-12T04:05:12.330081+0000', 'last_active': '2026-02-12T04:05:12.330081+0000', 'last_peered': '2026-02-12T04:05:12.330081+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:12.330081+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:18.954 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:19.175 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:19.175 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":642,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:19.184 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:19.184 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:19.365 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:19.366 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:19.374 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2026-02-12T04:05:17.738979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001585"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:22.376 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:22.553 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:22.553 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:22.561 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2026-02-12T04:05:21.739531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001601"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:22.562 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:22.740 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:22.741 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:22.750 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2026-02-12T04:05:21.739531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001601"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:22.751 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:22.751 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73244, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:12.330081+0000', 'last_change': '2026-02-12T04:05:12.330081+0000', 'last_active': '2026-02-12T04:05:12.330081+0000', 'last_peered': '2026-02-12T04:05:12.330081+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:12.330081+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:22.751 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:22.971 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:22.971 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":646,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:22.981 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:22.981 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:23.160 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:23.160 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:23.169 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2026-02-12T04:05:21.739531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001601"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:26.171 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:26.348 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:26.348 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:26.357 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2026-02-12T04:05:25.740145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001716"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:26.358 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:26.537 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:26.537 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:26.546 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2026-02-12T04:05:25.740145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001716"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:26.547 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:26.547 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73244, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:12.330081+0000', 'last_change': '2026-02-12T04:05:12.330081+0000', 'last_active': '2026-02-12T04:05:12.330081+0000', 'last_peered': '2026-02-12T04:05:12.330081+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:12.330081+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:26.547 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:26.769 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:26.769 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":650,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-12T04:03:17.723921+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:26.778 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:26.779 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:26.958 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:26.958 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:26.967 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2026-02-12T04:05:25.740145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001716"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:29.969 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:30.159 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:30.159 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:30.168 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2026-02-12T04:05:29.740605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001625"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:30.169 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:30.348 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:30.348 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:30.357 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2026-02-12T04:05:29.740605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001625"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:30.358 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:30.359 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73244, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:12.330081+0000', 'last_change': '2026-02-12T04:05:12.330081+0000', 'last_active': '2026-02-12T04:05:12.330081+0000', 'last_peered': '2026-02-12T04:05:12.330081+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:12.330081+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:30.359 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:30.577 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:30.577 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":654,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:30.586 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:30.586 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:30.761 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:30.762 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:30.770 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2026-02-12T04:05:29.740605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001625"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:33.771 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:33.951 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:33.951 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:33.960 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2026-02-12T04:05:33.741181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001649"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986263,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:33.961 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:34.143 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:34.144 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:34.153 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2026-02-12T04:05:33.741181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001649"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986263,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:34.154 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:34.154 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73244, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:12.330081+0000', 'last_change': '2026-02-12T04:05:12.330081+0000', 'last_active': '2026-02-12T04:05:12.330081+0000', 'last_peered': '2026-02-12T04:05:12.330081+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:12.330081+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:34.154 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:34.374 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:34.374 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":657,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:34.384 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:34.384 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:34.559 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:34.560 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:34.568 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2026-02-12T04:05:33.741181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001649"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986263,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:37.570 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:37.746 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:37.746 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:37.755 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2026-02-12T04:05:37.741769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001624"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:37.756 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:37.932 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:37.932 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:37.941 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2026-02-12T04:05:37.741769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001624"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:37.942 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:37.942 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73244, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:12.330081+0000', 'last_change': '2026-02-12T04:05:12.330081+0000', 'last_active': '2026-02-12T04:05:12.330081+0000', 'last_peered': '2026-02-12T04:05:12.330081+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:12.330081+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:37.943 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:38.163 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:38.163 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":661,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:38.172 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:38.172 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:38.349 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:38.350 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:38.359 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2026-02-12T04:05:37.741769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001624"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:41.361 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:41.540 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:41.540 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:41.549 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2026-02-12T04:05:39.742004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001637"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:41.550 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:41.729 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:41.729 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:41.738 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2026-02-12T04:05:39.742004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001637"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:41.739 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:41.739 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73244, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:12.330081+0000', 'last_change': '2026-02-12T04:05:12.330081+0000', 'last_active': '2026-02-12T04:05:12.330081+0000', 'last_peered': '2026-02-12T04:05:12.330081+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:12.330081+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:41.740 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:41.966 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:41.967 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":665,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:41.976 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:41.977 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:42.247 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:05:42.245+0000 7f81c5d74640 0 --2- 10.20.193.103:0/504915204 >> [v2:10.20.193.103:3302/0,v1:10.20.193.103:6791/0] conn(0x7f81c8133a90 0x7f81c811e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:05:42.357 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:42.358 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:42.366 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2026-02-12T04:05:41.742255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001650"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73244,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:12.330081+0000","last_change":"2026-02-12T04:05:12.330081+0000","last_active":"2026-02-12T04:05:12.330081+0000","last_peered":"2026-02-12T04:05:12.330081+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:12.330081+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:45.368 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:45.549 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:45.550 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:45.558 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2026-02-12T04:05:43.742511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001620"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:45.559 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:45.736 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:45.736 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:45.745 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2026-02-12T04:05:43.742511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001620"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:45.746 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:45.746 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73248, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:42.330571+0000', 'last_change': '2026-02-12T04:05:42.330571+0000', 'last_active': '2026-02-12T04:05:42.330571+0000', 'last_peered': '2026-02-12T04:05:42.330571+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:42.330571+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:45.746 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:45.965 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:45.966 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":669,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:45.975 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:45.975 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:46.152 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:46.152 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:46.161 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2026-02-12T04:05:45.742818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001637"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986266,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:49.162 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:49.343 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:49.344 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:49.352 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2026-02-12T04:05:47.743052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001606"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986266,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:49.353 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:49.531 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:49.531 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:49.540 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2026-02-12T04:05:47.743052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001606"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986266,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:49.541 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:49.541 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73248, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:42.330571+0000', 'last_change': '2026-02-12T04:05:42.330571+0000', 'last_active': '2026-02-12T04:05:42.330571+0000', 'last_peered': '2026-02-12T04:05:42.330571+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:42.330571+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:49.541 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:49.760 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:49.760 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":673,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:49.769 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:49.770 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:49.947 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:49.948 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:49.957 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2026-02-12T04:05:49.743347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001577"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986266,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:52.960 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:53.137 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:53.137 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:53.146 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2026-02-12T04:05:51.743611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001606"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986267,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:53.147 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:53.327 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:53.328 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:53.337 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2026-02-12T04:05:51.743611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001606"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986267,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:53.338 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:53.338 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73248, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:42.330571+0000', 'last_change': '2026-02-12T04:05:42.330571+0000', 'last_active': '2026-02-12T04:05:42.330571+0000', 'last_peered': '2026-02-12T04:05:42.330571+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:42.330571+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:53.338 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:53.558 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:53.559 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":677,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:53.568 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:53.569 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:53.749 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:53.749 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:53.758 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2026-02-12T04:05:53.743924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001668"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986267,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:56.761 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:56.939 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:56.939 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:56.949 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2026-02-12T04:05:55.744158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001647"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986268,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:56.950 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:57.128 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:57.128 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:57.137 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2026-02-12T04:05:55.744158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001647"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986268,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:05:57.138 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:05:57.138 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73248, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:42.330571+0000', 'last_change': '2026-02-12T04:05:42.330571+0000', 'last_active': '2026-02-12T04:05:42.330571+0000', 'last_peered': '2026-02-12T04:05:42.330571+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:42.330571+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:05:57.139 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:05:57.367 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:57.367 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":680,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:05:57.377 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:05:57.377 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:05:57.555 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:05:57.556 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:05:57.565 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2026-02-12T04:05:55.744158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001647"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986268,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:00.567 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:00.749 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:00.749 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:00.759 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2026-02-12T04:05:59.744588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001536"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986268,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:00.760 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:00.943 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:00.943 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:00.952 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2026-02-12T04:05:59.744588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001536"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986268,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:00.953 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:00.954 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73248, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:42.330571+0000', 'last_change': '2026-02-12T04:05:42.330571+0000', 'last_active': '2026-02-12T04:05:42.330571+0000', 'last_peered': '2026-02-12T04:05:42.330571+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:42.330571+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:00.954 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:01.173 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:01.173 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":684,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:06:01.182 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:01.183 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:01.364 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:01.364 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:01.372 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2026-02-12T04:05:59.744588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001536"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986268,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:04.375 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:04.554 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:04.554 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:04.563 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2026-02-12T04:06:03.745110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001498"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986269,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:04.564 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:04.742 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:04.742 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:04.751 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2026-02-12T04:06:03.745110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001498"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986269,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:04.752 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:04.752 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73248, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:42.330571+0000', 'last_change': '2026-02-12T04:05:42.330571+0000', 'last_active': '2026-02-12T04:05:42.330571+0000', 'last_peered': '2026-02-12T04:05:42.330571+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:42.330571+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:04.753 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:04.973 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:04.973 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":688,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:06:04.983 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:04.983 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:05.160 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:05.160 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:05.168 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2026-02-12T04:06:03.745110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001498"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986269,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:08.171 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:08.350 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:08.351 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:08.359 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2026-02-12T04:06:07.745591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001433"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986270,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:08.360 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:08.534 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:08.535 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:08.543 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2026-02-12T04:06:07.745591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001433"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986270,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:08.543 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:08.543 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73248, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:42.330571+0000', 'last_change': '2026-02-12T04:05:42.330571+0000', 'last_active': '2026-02-12T04:05:42.330571+0000', 'last_peered': '2026-02-12T04:05:42.330571+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:42.330571+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:08.543 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:08.760 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:08.760 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":692,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:06:08.769 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:08.769 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:08.946 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:08.947 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:08.956 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2026-02-12T04:06:07.745591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001433"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986270,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:11.958 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:12.140 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:12.140 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:12.149 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2026-02-12T04:06:11.746188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986271,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:12.150 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:12.325 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:12.325 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:12.333 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2026-02-12T04:06:11.746188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986271,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:12.334 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:12.334 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73248, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:05:42.330571+0000', 'last_change': '2026-02-12T04:05:42.330571+0000', 'last_active': '2026-02-12T04:05:42.330571+0000', 'last_peered': '2026-02-12T04:05:42.330571+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:05:42.330571+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:12.335 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:12.555 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:12.556 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":696,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:06:12.565 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:12.565 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:12.744 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:12.744 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:12.753 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2026-02-12T04:06:11.746188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73248,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:05:42.330571+0000","last_change":"2026-02-12T04:05:42.330571+0000","last_active":"2026-02-12T04:05:42.330571+0000","last_peered":"2026-02-12T04:05:42.330571+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:05:42.330571+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986271,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:15.755 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:15.933 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:15.934 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:15.943 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2026-02-12T04:06:15.746796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001685"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986272,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:15.944 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:16.121 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:16.121 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:16.130 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2026-02-12T04:06:15.746796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001685"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986272,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:16.131 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:16.131 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73252, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:12.331087+0000', 'last_change': '2026-02-12T04:06:12.331087+0000', 'last_active': '2026-02-12T04:06:12.331087+0000', 'last_peered': '2026-02-12T04:06:12.331087+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:12.331087+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:16.132 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:16.353 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:16.353 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":699,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:06:16.363 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:16.364 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:16.546 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:16.547 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:16.556 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2026-02-12T04:06:15.746796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001685"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986272,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:19.558 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:19.740 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:19.740 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:19.749 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2026-02-12T04:06:17.747053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001702"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986272,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:19.751 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:20.020 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:06:20.019+0000 7fefe1673640 0 --2- 10.20.193.103:0/3107148500 >> v2:10.20.193.103:3300/0 conn(0x7fefdc133f50 0x7fefdc134320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:06:20.129 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:20.129 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:20.138 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2026-02-12T04:06:19.747284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001692"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986272,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:20.139 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:20.139 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73252, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:12.331087+0000', 'last_change': '2026-02-12T04:06:12.331087+0000', 'last_active': '2026-02-12T04:06:12.331087+0000', 'last_peered': '2026-02-12T04:06:12.331087+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:12.331087+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:20.140 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:20.361 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:20.361 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":703,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:06:20.371 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:20.371 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:20.547 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:20.548 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:20.557 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2026-02-12T04:06:19.747284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001692"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986272,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:23.559 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:23.739 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:23.739 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:23.748 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2026-02-12T04:06:21.747549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001680"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986273,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:23.749 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:24.021 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:24.022 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:24.031 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2026-02-12T04:06:23.747859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001671"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986273,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:24.032 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:24.033 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73252, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:12.331087+0000', 'last_change': '2026-02-12T04:06:12.331087+0000', 'last_active': '2026-02-12T04:06:12.331087+0000', 'last_peered': '2026-02-12T04:06:12.331087+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:12.331087+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:24.033 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:24.255 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:24.256 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":707,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:06:24.265 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:24.266 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:24.445 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:24.445 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:24.454 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2026-02-12T04:06:23.747859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001671"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986273,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:27.456 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:27.635 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:27.636 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:27.644 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2026-02-12T04:06:25.748122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001630"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986274,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:27.646 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:27.822 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:27.823 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:27.831 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2026-02-12T04:06:27.748458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001662"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986274,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:27.832 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:27.833 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73252, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:12.331087+0000', 'last_change': '2026-02-12T04:06:12.331087+0000', 'last_active': '2026-02-12T04:06:12.331087+0000', 'last_peered': '2026-02-12T04:06:12.331087+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:12.331087+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:27.833 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:28.061 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:28.062 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":711,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:06:28.071 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:28.071 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:28.256 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:28.257 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:28.265 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2026-02-12T04:06:27.748458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001662"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986274,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:31.267 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:31.444 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:31.445 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:31.453 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2026-02-12T04:06:29.748704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001651"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986274,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:31.455 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:31.633 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:31.634 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:31.643 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2026-02-12T04:06:29.748704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001651"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986274,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:31.644 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:31.644 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73252, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:12.331087+0000', 'last_change': '2026-02-12T04:06:12.331087+0000', 'last_active': '2026-02-12T04:06:12.331087+0000', 'last_peered': '2026-02-12T04:06:12.331087+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:12.331087+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:31.644 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:31.864 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:31.864 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":715,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:06:31.873 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:31.874 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:32.052 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:32.053 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:32.062 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2026-02-12T04:06:31.749044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001760"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986275,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:35.063 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:35.242 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:35.243 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:35.252 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2026-02-12T04:06:33.749321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001771"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986275,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:35.253 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:35.432 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:35.432 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:35.442 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2026-02-12T04:06:33.749321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001771"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986275,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:35.443 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:35.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73252, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:12.331087+0000', 'last_change': '2026-02-12T04:06:12.331087+0000', 'last_active': '2026-02-12T04:06:12.331087+0000', 'last_peered': '2026-02-12T04:06:12.331087+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:12.331087+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:35.443 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:35.663 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:35.663 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":719,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:06:35.672 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:35.672 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:35.849 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:35.849 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:35.858 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2026-02-12T04:06:35.749665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001805"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986276,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:38.860 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:39.044 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:39.045 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:39.054 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2026-02-12T04:06:37.749894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001772"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986276,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:39.056 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:39.233 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:39.233 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:39.242 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2026-02-12T04:06:37.749894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001772"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986276,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:39.243 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:39.243 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73252, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:12.331087+0000', 'last_change': '2026-02-12T04:06:12.331087+0000', 'last_active': '2026-02-12T04:06:12.331087+0000', 'last_peered': '2026-02-12T04:06:12.331087+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:12.331087+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:39.243 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:39.472 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:39.472 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":722,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:06:39.481 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:39.481 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:39.655 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:39.656 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:39.664 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2026-02-12T04:06:37.749894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001772"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:02:40.226590+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:02:40.226590+0000","last_peered":"2026-02-12T04:02:40.226590+0000","last_clean":"2026-02-12T04:02:40.226590+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:02:40.226590+0000","last_undegraded":"2026-02-12T04:02:40.226590+0000","last_fullsized":"2026-02-12T04:02:40.226590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986276,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:42.666 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:42.846 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:42.846 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:42.855 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2026-02-12T04:06:41.750391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001687"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986277,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:42.856 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:43.031 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:43.032 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:43.041 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2026-02-12T04:06:41.750391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001687"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986277,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:43.042 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:43.042 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73252, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:12.331087+0000', 'last_change': '2026-02-12T04:06:12.331087+0000', 'last_active': '2026-02-12T04:06:12.331087+0000', 'last_peered': '2026-02-12T04:06:12.331087+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:12.331087+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:43.042 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:43.260 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:43.260 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":726,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:06:43.270 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:43.270 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:43.448 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:43.448 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:43.457 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2026-02-12T04:06:41.750391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001687"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73252,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:12.331087+0000","last_change":"2026-02-12T04:06:12.331087+0000","last_active":"2026-02-12T04:06:12.331087+0000","last_peered":"2026-02-12T04:06:12.331087+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:12.331087+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986277,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:46.459 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:46.639 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:46.640 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:46.649 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2026-02-12T04:06:45.751009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001688"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73256,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331511+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331511+0000","last_peered":"2026-02-12T04:06:42.331511+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331511+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986278,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:46.650 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:46.830 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:46.830 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:46.839 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2026-02-12T04:06:45.751009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001688"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73256,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331511+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331511+0000","last_peered":"2026-02-12T04:06:42.331511+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331511+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986278,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:46.840 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:46.840 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73256, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:42.331511+0000', 'last_change': '2026-02-12T04:06:42.331511+0000', 'last_active': '2026-02-12T04:06:42.331511+0000', 'last_peered': '2026-02-12T04:06:42.331511+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:42.331511+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:46.841 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:47.058 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:47.059 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":730,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:06:47.067 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:47.068 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:47.244 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:47.245 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:47.253 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2026-02-12T04:06:45.751009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001688"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73256,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331511+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331511+0000","last_peered":"2026-02-12T04:06:42.331511+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331511+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986278,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:50.255 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:50.434 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:50.434 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:50.443 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2026-02-12T04:06:49.751563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001668"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73256,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331511+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331511+0000","last_peered":"2026-02-12T04:06:42.331511+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331511+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986278,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:50.444 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:50.622 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:50.623 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:50.631 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2026-02-12T04:06:49.751563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001668"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73256,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331511+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331511+0000","last_peered":"2026-02-12T04:06:42.331511+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331511+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986278,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:50.632 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:50.633 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73256, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:42.331511+0000', 'last_change': '2026-02-12T04:06:42.331511+0000', 'last_active': '2026-02-12T04:06:42.331511+0000', 'last_peered': '2026-02-12T04:06:42.331511+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:42.331511+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:50.633 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:50.853 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:50.853 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":734,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-12T04:05:27.740392+0000","services":{}},"progress_events":{}} 2026-02-12T04:06:50.863 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:50.863 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:51.041 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:51.041 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:51.050 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2026-02-12T04:06:49.751563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001668"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73256,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331511+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331511+0000","last_peered":"2026-02-12T04:06:42.331511+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331511+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986278,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:54.052 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:54.232 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:54.232 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:54.241 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2026-02-12T04:06:53.752061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001669"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986279,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:54.242 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:54.422 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:54.423 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:54.432 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2026-02-12T04:06:53.752061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001669"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986279,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:54.433 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:54.433 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73257, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:42.331537+0000', 'last_change': '2026-02-12T04:06:42.331511+0000', 'last_active': '2026-02-12T04:06:42.331537+0000', 'last_peered': '2026-02-12T04:06:42.331537+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:42.331537+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:54.434 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:54.658 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:54.658 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":738,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-12T04:06:53.752096+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:06:54.667 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:54.667 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:54.846 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:54.846 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:54.855 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2026-02-12T04:06:53.752061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001669"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986279,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:57.857 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:58.037 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:58.038 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:58.046 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2026-02-12T04:06:57.752609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986280,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:58.047 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:58.226 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:58.227 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:58.235 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2026-02-12T04:06:57.752609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986280,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:06:58.236 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:06:58.236 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73257, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:42.331537+0000', 'last_change': '2026-02-12T04:06:42.331511+0000', 'last_active': '2026-02-12T04:06:42.331537+0000', 'last_peered': '2026-02-12T04:06:42.331537+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:42.331537+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:06:58.237 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:06:58.461 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:58.461 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":741,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-12T04:06:53.752096+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:06:58.471 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:06:58.472 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:06:58.659 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:06:58.660 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:06:58.669 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2026-02-12T04:06:57.752609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986280,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:01.670 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:01.850 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:01.851 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:01.860 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2026-02-12T04:07:01.753146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001583"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986281,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:01.861 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:02.038 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:02.038 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:02.046 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2026-02-12T04:07:01.753146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001583"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986281,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:02.047 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:02.047 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73257, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:42.331537+0000', 'last_change': '2026-02-12T04:06:42.331511+0000', 'last_active': '2026-02-12T04:06:42.331537+0000', 'last_peered': '2026-02-12T04:06:42.331537+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:42.331537+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:02.047 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:02.317 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:07:02.316+0000 7f2a25be4640 0 --2- 10.20.193.103:0/2755355201 >> v2:10.20.193.103:3300/0 conn(0x7f2a2011acd0 0x7f2a20195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:07:02.519 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:07:02.518+0000 7f2a253e3640 0 --2- 10.20.193.103:0/2755355201 >> [v2:10.20.193.103:3302/0,v1:10.20.193.103:6791/0] conn(0x7f2a08003890 0x7f2a08003c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:07:02.669 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:02.669 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":746,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-12T04:06:53.752096+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:07:02.678 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:02.678 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:02.858 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:02.858 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:02.867 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2026-02-12T04:07:01.753146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001583"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986281,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:05.869 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:06.049 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:06.050 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:06.058 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2026-02-12T04:07:05.753714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001652"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986282,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:06.060 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:06.238 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:06.238 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:06.247 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2026-02-12T04:07:05.753714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001652"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986282,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:06.248 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:06.248 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73257, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:42.331537+0000', 'last_change': '2026-02-12T04:06:42.331511+0000', 'last_active': '2026-02-12T04:06:42.331537+0000', 'last_peered': '2026-02-12T04:06:42.331537+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:42.331537+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:06.248 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:06.467 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:06.467 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":749,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-12T04:06:53.752096+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:07:06.477 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:06.477 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:06.652 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:06.653 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:06.661 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2026-02-12T04:07:05.753714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001652"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986282,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:09.662 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:09.847 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:09.848 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:09.856 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2026-02-12T04:07:09.754273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001663"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986282,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:09.857 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:10.034 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:10.035 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:10.043 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2026-02-12T04:07:09.754273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001663"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986282,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:10.044 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:10.044 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73257, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:06:42.331537+0000', 'last_change': '2026-02-12T04:06:42.331511+0000', 'last_active': '2026-02-12T04:06:42.331537+0000', 'last_peered': '2026-02-12T04:06:42.331537+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:06:42.331537+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:10.044 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:10.261 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:10.262 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":753,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-12T04:06:53.752096+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:07:10.271 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:10.272 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:10.451 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:10.451 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:10.460 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2026-02-12T04:07:09.754273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001663"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986282,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:13.462 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:13.639 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:13.639 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:13.647 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2026-02-12T04:07:11.754518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001659"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73257,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:06:42.331537+0000","last_change":"2026-02-12T04:06:42.331511+0000","last_active":"2026-02-12T04:06:42.331537+0000","last_peered":"2026-02-12T04:06:42.331537+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:06:42.331537+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986283,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:13.649 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:13.827 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:13.828 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:13.836 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2026-02-12T04:07:13.754841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001694"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986283,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:13.837 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:13.837 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73261, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:12.331982+0000', 'last_change': '2026-02-12T04:07:12.331982+0000', 'last_active': '2026-02-12T04:07:12.331982+0000', 'last_peered': '2026-02-12T04:07:12.331982+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:12.331982+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:13.838 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:14.058 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:14.058 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":757,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-12T04:06:53.752096+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:07:14.068 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:14.068 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:14.250 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:14.251 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:14.259 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2026-02-12T04:07:13.754841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001694"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986283,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:17.261 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:17.441 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:17.442 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:17.451 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2026-02-12T04:07:15.755081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001666"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986284,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:17.452 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:17.631 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:17.632 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:17.640 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2026-02-12T04:07:15.755081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001666"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986284,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:17.641 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:17.641 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73261, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:12.331982+0000', 'last_change': '2026-02-12T04:07:12.331982+0000', 'last_active': '2026-02-12T04:07:12.331982+0000', 'last_peered': '2026-02-12T04:07:12.331982+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:12.331982+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:17.642 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:17.863 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:17.863 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":761,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-12T04:06:53.752096+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:07:17.873 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:17.873 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:18.054 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:18.055 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:18.063 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2026-02-12T04:07:17.755333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001619"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986284,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:21.065 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:21.245 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:21.246 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:21.255 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2026-02-12T04:07:19.755590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001604"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986284,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:21.256 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:21.435 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:21.436 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:21.445 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2026-02-12T04:07:19.755590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001604"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986284,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:21.446 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:21.447 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73261, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:12.331982+0000', 'last_change': '2026-02-12T04:07:12.331982+0000', 'last_active': '2026-02-12T04:07:12.331982+0000', 'last_peered': '2026-02-12T04:07:12.331982+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:12.331982+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:21.447 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:21.665 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:21.665 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":765,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-12T04:07:17.755365+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:07:21.675 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:21.676 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:21.851 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:21.851 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:21.860 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2026-02-12T04:07:21.755844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001571"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986285,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:24.863 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:25.040 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:25.040 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:25.049 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2026-02-12T04:07:23.756076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001557"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986285,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:25.050 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:25.231 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:25.231 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:25.240 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2026-02-12T04:07:23.756076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001557"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986285,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:25.241 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:25.241 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73261, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:12.331982+0000', 'last_change': '2026-02-12T04:07:12.331982+0000', 'last_active': '2026-02-12T04:07:12.331982+0000', 'last_peered': '2026-02-12T04:07:12.331982+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:12.331982+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:25.242 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:25.465 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:25.466 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":768,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:07:25.476 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:25.476 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:25.748 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:07:25.747+0000 7fcc17fff640 0 --2- 10.20.193.103:0/1436152698 >> [v2:10.20.193.103:3301/0,v1:10.20.193.103:6790/0] conn(0x7fcc18131020 0x7fcc18181af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:07:25.857 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:25.858 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:25.867 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2026-02-12T04:07:25.756353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001511"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986286,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:28.869 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:29.051 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:29.052 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:29.061 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2026-02-12T04:07:27.756593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001512"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986286,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:29.062 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:29.243 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:29.244 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:29.252 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2026-02-12T04:07:27.756593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001512"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986286,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:29.253 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:29.253 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73261, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:12.331982+0000', 'last_change': '2026-02-12T04:07:12.331982+0000', 'last_active': '2026-02-12T04:07:12.331982+0000', 'last_peered': '2026-02-12T04:07:12.331982+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:12.331982+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:29.254 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:29.479 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:29.480 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":772,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:07:29.490 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:29.490 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:29.670 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:29.671 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:29.679 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2026-02-12T04:07:27.756593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001512"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986286,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:32.681 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:33.755 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:33.756 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:33.765 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2026-02-12T04:07:31.757173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001582"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986287,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:33.766 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:33.943 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:33.943 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:33.952 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2026-02-12T04:07:33.757493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001649"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986287,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:33.952 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:33.953 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73261, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:12.331982+0000', 'last_change': '2026-02-12T04:07:12.331982+0000', 'last_active': '2026-02-12T04:07:12.331982+0000', 'last_peered': '2026-02-12T04:07:12.331982+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:12.331982+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:33.953 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:34.173 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:34.174 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":777,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:07:34.183 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:34.183 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:34.362 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:34.363 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:34.371 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2026-02-12T04:07:33.757493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001649"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986287,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:37.373 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:37.554 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:37.554 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:37.563 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2026-02-12T04:07:35.757699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001623"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986288,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:37.564 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:37.838 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:37.838 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:37.848 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2026-02-12T04:07:37.758021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001667"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986288,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:37.849 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:37.849 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73261, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:12.331982+0000', 'last_change': '2026-02-12T04:07:12.331982+0000', 'last_active': '2026-02-12T04:07:12.331982+0000', 'last_peered': '2026-02-12T04:07:12.331982+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:12.331982+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:37.850 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:38.073 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:38.074 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":781,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:07:38.083 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:38.083 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:38.260 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:38.261 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:38.269 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2026-02-12T04:07:37.758021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001667"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986288,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:41.271 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:41.450 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:41.450 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:41.459 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2026-02-12T04:07:39.758256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001662"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986288,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:41.460 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:41.644 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:41.644 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:41.653 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2026-02-12T04:07:39.758256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001662"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986288,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:41.654 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:41.654 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73261, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:12.331982+0000', 'last_change': '2026-02-12T04:07:12.331982+0000', 'last_active': '2026-02-12T04:07:12.331982+0000', 'last_peered': '2026-02-12T04:07:12.331982+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:12.331982+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:41.655 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:41.876 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:41.876 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":785,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:07:41.886 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:41.886 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:42.064 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:42.064 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:42.073 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2026-02-12T04:07:41.758551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001686"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73261,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:12.331982+0000","last_change":"2026-02-12T04:07:12.331982+0000","last_active":"2026-02-12T04:07:12.331982+0000","last_peered":"2026-02-12T04:07:12.331982+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:12.331982+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986289,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:45.074 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:45.257 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:45.258 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:45.266 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2026-02-12T04:07:43.758781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001608"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986289,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:45.268 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:45.443 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:45.444 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:45.452 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2026-02-12T04:07:43.758781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001608"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986289,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:45.453 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:45.454 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73265, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:42.332448+0000', 'last_change': '2026-02-12T04:07:42.332448+0000', 'last_active': '2026-02-12T04:07:42.332448+0000', 'last_peered': '2026-02-12T04:07:42.332448+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:42.332448+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:45.454 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:45.679 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:45.680 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":789,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:07:45.689 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:45.689 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:45.866 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:45.866 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:45.876 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2026-02-12T04:07:45.759016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001522"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986290,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:48.877 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:49.056 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:49.057 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:49.066 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2026-02-12T04:07:47.759228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001528"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986290,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:49.068 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:49.245 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:49.245 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:49.253 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2026-02-12T04:07:47.759228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001528"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986290,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:49.254 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:49.255 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73265, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:42.332448+0000', 'last_change': '2026-02-12T04:07:42.332448+0000', 'last_active': '2026-02-12T04:07:42.332448+0000', 'last_peered': '2026-02-12T04:07:42.332448+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:42.332448+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:49.255 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:49.489 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:49.489 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":792,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:07:49.499 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:49.499 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:49.679 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:49.679 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:49.688 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2026-02-12T04:07:47.759228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001528"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986290,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:52.690 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:52.869 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:52.870 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:52.879 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2026-02-12T04:07:51.759766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001509"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986291,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:52.880 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:53.056 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:53.057 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:53.065 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2026-02-12T04:07:51.759766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001509"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986291,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:53.066 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:53.066 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73265, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:42.332448+0000', 'last_change': '2026-02-12T04:07:42.332448+0000', 'last_active': '2026-02-12T04:07:42.332448+0000', 'last_peered': '2026-02-12T04:07:42.332448+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:42.332448+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:53.066 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:53.287 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:53.287 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":796,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:07:53.296 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:53.297 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:53.476 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:53.476 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:53.486 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2026-02-12T04:07:51.759766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001509"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986291,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:56.488 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:56.666 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:56.666 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:56.674 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2026-02-12T04:07:55.760377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001595"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:56.675 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:56.947 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:07:56.946+0000 7f556ed76640 0 --2- 10.20.193.103:0/2308933120 >> [v2:10.20.193.103:3301/0,v1:10.20.193.103:6790/0] conn(0x7f5570133f50 0x7f557012b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:07:57.055 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:57.056 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:57.064 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2026-02-12T04:07:55.760377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001595"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:07:57.065 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:07:57.065 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73265, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:42.332448+0000', 'last_change': '2026-02-12T04:07:42.332448+0000', 'last_active': '2026-02-12T04:07:42.332448+0000', 'last_peered': '2026-02-12T04:07:42.332448+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:42.332448+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:07:57.066 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:07:57.285 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:57.286 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":800,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:07:57.295 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:07:57.296 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:07:57.474 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:07:57.474 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:07:57.483 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2026-02-12T04:07:55.760377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001595"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:00.485 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:01.560 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:01.560 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:01.570 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2026-02-12T04:07:59.760842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001613"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:01.572 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:01.746 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:01.747 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:01.755 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2026-02-12T04:07:59.760842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001613"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:01.756 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:01.756 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73265, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:42.332448+0000', 'last_change': '2026-02-12T04:07:42.332448+0000', 'last_active': '2026-02-12T04:07:42.332448+0000', 'last_peered': '2026-02-12T04:07:42.332448+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:42.332448+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:01.757 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:01.980 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:01.980 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":805,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:01.991 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:01.991 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:02.167 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:02.168 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:02.175 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2026-02-12T04:08:01.761090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001626"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:05.177 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:05.357 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:05.358 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:05.367 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2026-02-12T04:08:03.761332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001566"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:05.368 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:05.543 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:05.544 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:05.553 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2026-02-12T04:08:03.761332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001566"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:05.554 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:05.554 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73265, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:42.332448+0000', 'last_change': '2026-02-12T04:07:42.332448+0000', 'last_active': '2026-02-12T04:07:42.332448+0000', 'last_peered': '2026-02-12T04:07:42.332448+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:42.332448+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:05.554 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:05.777 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:05.777 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":809,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:05.786 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:05.787 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:05.964 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:05.965 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:05.973 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2026-02-12T04:08:05.761640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001571"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986294,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:08.975 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:09.160 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:09.160 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:09.169 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2026-02-12T04:08:07.761927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001550"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986294,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:09.170 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:09.346 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:09.347 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:09.355 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2026-02-12T04:08:07.761927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001550"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986294,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:09.356 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:09.357 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73265, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:42.332448+0000', 'last_change': '2026-02-12T04:07:42.332448+0000', 'last_active': '2026-02-12T04:07:42.332448+0000', 'last_peered': '2026-02-12T04:07:42.332448+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:42.332448+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:09.357 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:09.576 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:09.576 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":813,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:09.586 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:09.586 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:09.761 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:09.761 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:09.769 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2026-02-12T04:08:07.761927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001550"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986294,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:12.771 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:12.949 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:12.950 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:12.958 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2026-02-12T04:08:11.762507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001665"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986295,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:12.959 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:13.135 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:13.136 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:13.144 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2026-02-12T04:08:11.762507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001665"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986295,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:13.145 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:13.145 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73265, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:07:42.332448+0000', 'last_change': '2026-02-12T04:07:42.332448+0000', 'last_active': '2026-02-12T04:07:42.332448+0000', 'last_peered': '2026-02-12T04:07:42.332448+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:07:42.332448+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:13.146 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:13.374 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:13.374 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":816,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:13.383 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:13.383 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:13.561 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:13.561 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:13.570 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2026-02-12T04:08:11.762507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001665"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73265,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:07:42.332448+0000","last_change":"2026-02-12T04:07:42.332448+0000","last_active":"2026-02-12T04:07:42.332448+0000","last_peered":"2026-02-12T04:07:42.332448+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:07:42.332448+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986295,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:16.572 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:16.751 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:16.752 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:16.761 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2026-02-12T04:08:15.763089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001757"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986296,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:16.762 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:16.939 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:16.940 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:16.948 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2026-02-12T04:08:15.763089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001757"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986296,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:16.949 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:16.949 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73269, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:12.333008+0000', 'last_change': '2026-02-12T04:08:12.333008+0000', 'last_active': '2026-02-12T04:08:12.333008+0000', 'last_peered': '2026-02-12T04:08:12.333008+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:12.333008+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:16.950 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:17.171 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:17.171 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":820,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:17.181 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:17.181 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:17.361 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:17.362 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:17.371 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2026-02-12T04:08:15.763089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001757"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986296,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:20.373 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:20.555 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:20.555 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:20.565 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2026-02-12T04:08:19.763628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986296,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:20.566 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:20.746 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:20.746 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:20.755 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2026-02-12T04:08:19.763628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986296,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:20.756 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:20.756 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73269, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:12.333008+0000', 'last_change': '2026-02-12T04:08:12.333008+0000', 'last_active': '2026-02-12T04:08:12.333008+0000', 'last_peered': '2026-02-12T04:08:12.333008+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:12.333008+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:20.757 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:20.978 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:20.978 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":824,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:20.988 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:20.988 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:21.164 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:21.164 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:21.173 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2026-02-12T04:08:19.763628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986296,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:24.175 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:24.357 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:24.357 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:24.366 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":439,"stamp":"2026-02-12T04:08:23.764074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001566"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986297,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:24.368 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:24.549 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:24.550 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:24.559 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":439,"stamp":"2026-02-12T04:08:23.764074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001566"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986297,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:24.559 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:24.559 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73269, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:12.333008+0000', 'last_change': '2026-02-12T04:08:12.333008+0000', 'last_active': '2026-02-12T04:08:12.333008+0000', 'last_peered': '2026-02-12T04:08:12.333008+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:12.333008+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:24.560 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:24.781 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:24.781 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":828,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:24.789 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:24.790 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:24.967 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:24.967 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:24.976 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":439,"stamp":"2026-02-12T04:08:23.764074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001566"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986297,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:27.977 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:28.157 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:28.157 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:28.166 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2026-02-12T04:08:27.764665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001575"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986298,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:28.167 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:28.345 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:28.345 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:28.354 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2026-02-12T04:08:27.764665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001575"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986298,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:28.355 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:28.355 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73269, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:12.333008+0000', 'last_change': '2026-02-12T04:08:12.333008+0000', 'last_active': '2026-02-12T04:08:12.333008+0000', 'last_peered': '2026-02-12T04:08:12.333008+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:12.333008+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:28.355 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:28.577 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:28.577 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":832,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:28.588 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:28.588 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:28.767 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:28.768 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:28.777 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2026-02-12T04:08:27.764665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001575"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986298,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:31.779 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:31.959 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:31.960 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:31.968 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2026-02-12T04:08:31.765145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986299,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:31.969 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:32.150 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:32.151 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:32.159 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2026-02-12T04:08:31.765145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986299,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:32.160 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:32.160 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73269, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:12.333008+0000', 'last_change': '2026-02-12T04:08:12.333008+0000', 'last_active': '2026-02-12T04:08:12.333008+0000', 'last_peered': '2026-02-12T04:08:12.333008+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:12.333008+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:32.161 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:32.384 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:32.384 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":835,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:32.393 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:32.394 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:32.570 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:32.571 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:32.580 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2026-02-12T04:08:31.765145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986299,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:35.581 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:35.760 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:35.760 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:35.769 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2026-02-12T04:08:33.765415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001549"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986299,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:35.769 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:35.948 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:35.948 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:35.957 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2026-02-12T04:08:35.765623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001549"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986300,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:35.957 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:35.957 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73269, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:12.333008+0000', 'last_change': '2026-02-12T04:08:12.333008+0000', 'last_active': '2026-02-12T04:08:12.333008+0000', 'last_peered': '2026-02-12T04:08:12.333008+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:12.333008+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:35.957 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:36.178 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:36.178 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":839,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:36.187 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:36.187 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:36.366 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:36.367 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:36.375 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2026-02-12T04:08:35.765623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001549"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986300,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:39.377 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:39.558 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:39.559 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:39.568 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2026-02-12T04:08:37.765853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001514"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986300,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:39.570 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:39.745 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:39.745 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:39.753 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2026-02-12T04:08:37.765853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001514"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986300,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:39.755 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:39.755 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73269, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:12.333008+0000', 'last_change': '2026-02-12T04:08:12.333008+0000', 'last_active': '2026-02-12T04:08:12.333008+0000', 'last_peered': '2026-02-12T04:08:12.333008+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:12.333008+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:39.755 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:39.978 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:39.978 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":843,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:39.988 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:39.989 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:40.165 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:40.166 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:40.174 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2026-02-12T04:08:39.766146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001481"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:04:40.236976+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:04:40.236976+0000","last_peered":"2026-02-12T04:04:40.236976+0000","last_clean":"2026-02-12T04:04:40.236976+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:04:40.236976+0000","last_undegraded":"2026-02-12T04:04:40.236976+0000","last_fullsized":"2026-02-12T04:04:40.236976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986300,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:43.176 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:43.355 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:43.356 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:43.364 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2026-02-12T04:08:41.766373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001465"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986301,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:43.365 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:43.541 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:43.542 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:43.551 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2026-02-12T04:08:41.766373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001465"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73269,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:12.333008+0000","last_change":"2026-02-12T04:08:12.333008+0000","last_active":"2026-02-12T04:08:12.333008+0000","last_peered":"2026-02-12T04:08:12.333008+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:12.333008+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986301,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:43.552 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:43.552 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73269, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:12.333008+0000', 'last_change': '2026-02-12T04:08:12.333008+0000', 'last_active': '2026-02-12T04:08:12.333008+0000', 'last_peered': '2026-02-12T04:08:12.333008+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:12.333008+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:43.553 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:43.774 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:43.774 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":847,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:43.783 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:43.783 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:43.961 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:43.962 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:43.970 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2026-02-12T04:08:43.766619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001473"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73273,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333425+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333425+0000","last_peered":"2026-02-12T04:08:42.333425+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333425+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986301,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:46.972 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:47.151 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:47.151 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:47.160 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2026-02-12T04:08:45.766864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001449"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73273,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333425+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333425+0000","last_peered":"2026-02-12T04:08:42.333425+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333425+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986302,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:47.161 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:47.338 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:47.338 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:47.347 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2026-02-12T04:08:45.766864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001449"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73273,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333425+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333425+0000","last_peered":"2026-02-12T04:08:42.333425+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333425+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986302,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:47.348 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:47.348 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73273, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:42.333425+0000', 'last_change': '2026-02-12T04:08:42.333425+0000', 'last_active': '2026-02-12T04:08:42.333425+0000', 'last_peered': '2026-02-12T04:08:42.333425+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:42.333425+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:47.349 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:47.568 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:47.568 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":851,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:47.577 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:47.577 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:47.757 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:47.758 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:47.767 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2026-02-12T04:08:45.766864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001449"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73273,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333425+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333425+0000","last_peered":"2026-02-12T04:08:42.333425+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333425+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986302,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:50.770 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:50.951 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:50.951 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:50.960 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2026-02-12T04:08:49.767308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001454"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73273,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333425+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333425+0000","last_peered":"2026-02-12T04:08:42.333425+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333425+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986302,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:50.961 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:51.234 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:51.234 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:51.243 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2026-02-12T04:08:49.767308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001454"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73273,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333425+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333425+0000","last_peered":"2026-02-12T04:08:42.333425+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333425+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986302,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:51.244 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:51.244 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73273, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:42.333425+0000', 'last_change': '2026-02-12T04:08:42.333425+0000', 'last_active': '2026-02-12T04:08:42.333425+0000', 'last_peered': '2026-02-12T04:08:42.333425+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:42.333425+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:51.245 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:51.464 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:51.465 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":854,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:51.474 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:51.474 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:51.654 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:51.655 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:51.664 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2026-02-12T04:08:49.767308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001454"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73273,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333425+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333425+0000","last_peered":"2026-02-12T04:08:42.333425+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333425+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986302,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:54.666 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:54.846 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:54.846 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:54.856 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2026-02-12T04:08:53.767771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001397"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:54.857 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:55.035 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:55.036 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:55.044 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2026-02-12T04:08:53.767771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001397"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:55.045 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:55.045 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73274, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:42.333452+0000', 'last_change': '2026-02-12T04:08:42.333425+0000', 'last_active': '2026-02-12T04:08:42.333452+0000', 'last_peered': '2026-02-12T04:08:42.333452+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:42.333452+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:55.046 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:55.267 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:55.267 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":858,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:55.277 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:55.277 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:55.457 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:55.457 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:55.468 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2026-02-12T04:08:53.767771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001397"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:58.469 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:58.651 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:58.651 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:58.660 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":456,"stamp":"2026-02-12T04:08:57.768321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986304,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:58.661 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:58.838 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:58.839 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:58.848 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":456,"stamp":"2026-02-12T04:08:57.768321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986304,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:08:58.849 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:08:58.849 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73274, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:42.333452+0000', 'last_change': '2026-02-12T04:08:42.333425+0000', 'last_active': '2026-02-12T04:08:42.333452+0000', 'last_peered': '2026-02-12T04:08:42.333452+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:42.333452+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:08:58.850 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:08:59.071 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:59.071 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":862,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:08:59.080 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:08:59.081 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:08:59.266 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:08:59.266 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:08:59.275 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":456,"stamp":"2026-02-12T04:08:57.768321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986304,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:02.277 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:02.455 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:02.455 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:02.464 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2026-02-12T04:09:01.768911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:02.465 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:02.646 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:02.647 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:02.656 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2026-02-12T04:09:01.768911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:02.656 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:02.656 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73274, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:42.333452+0000', 'last_change': '2026-02-12T04:08:42.333425+0000', 'last_active': '2026-02-12T04:08:42.333452+0000', 'last_peered': '2026-02-12T04:08:42.333452+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:42.333452+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:02.657 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:02.877 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:02.877 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":866,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:02.887 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:02.887 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:03.064 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:03.065 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:03.073 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2026-02-12T04:09:01.768911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:06.075 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:06.254 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:06.255 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:06.263 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2026-02-12T04:09:05.769483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001711"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:06.264 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:06.442 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:06.443 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:06.452 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2026-02-12T04:09:05.769483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001711"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:06.452 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:06.453 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73274, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:42.333452+0000', 'last_change': '2026-02-12T04:08:42.333425+0000', 'last_active': '2026-02-12T04:08:42.333452+0000', 'last_peered': '2026-02-12T04:08:42.333452+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:42.333452+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:06.453 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:06.672 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:06.672 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":870,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:06.682 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:06.682 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:06.860 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:06.860 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:06.869 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2026-02-12T04:09:05.769483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001711"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:09.870 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:10.051 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:10.052 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:10.061 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2026-02-12T04:09:09.770053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001732"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:10.062 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:10.239 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:10.240 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:10.248 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2026-02-12T04:09:09.770053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001732"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:10.250 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:10.250 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73274, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:08:42.333452+0000', 'last_change': '2026-02-12T04:08:42.333425+0000', 'last_active': '2026-02-12T04:08:42.333452+0000', 'last_peered': '2026-02-12T04:08:42.333452+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:08:42.333452+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:10.250 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:10.472 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:10.473 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":873,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:10.483 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:10.483 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:10.664 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:10.664 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:10.674 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2026-02-12T04:09:09.770053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001732"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73274,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:08:42.333452+0000","last_change":"2026-02-12T04:08:42.333425+0000","last_active":"2026-02-12T04:08:42.333452+0000","last_peered":"2026-02-12T04:08:42.333452+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:08:42.333452+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:13.675 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:13.854 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:13.855 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:13.865 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2026-02-12T04:09:13.770599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001688"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986307,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:13.866 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:14.044 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:14.044 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:14.053 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2026-02-12T04:09:13.770599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001688"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986307,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:14.054 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:14.055 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73278, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:12.334015+0000', 'last_change': '2026-02-12T04:09:12.334015+0000', 'last_active': '2026-02-12T04:09:12.334015+0000', 'last_peered': '2026-02-12T04:09:12.334015+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:12.334015+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:14.055 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:14.277 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:14.277 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":877,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:14.286 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:14.286 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:14.462 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:14.463 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:14.471 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2026-02-12T04:09:13.770599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001688"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986307,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:17.473 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:17.649 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:17.649 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:17.658 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2026-02-12T04:09:15.770865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001677"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986308,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:17.658 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:17.844 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:17.844 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:17.853 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2026-02-12T04:09:17.771154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001671"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986308,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:17.854 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:17.854 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73278, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:12.334015+0000', 'last_change': '2026-02-12T04:09:12.334015+0000', 'last_active': '2026-02-12T04:09:12.334015+0000', 'last_peered': '2026-02-12T04:09:12.334015+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:12.334015+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:17.854 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:18.074 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:18.075 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":881,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:18.084 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:18.085 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:18.355 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:09:18.354+0000 7f0b1bfff640 0 --2- 10.20.193.103:0/3013261708 >> [v2:10.20.193.103:3302/0,v1:10.20.193.103:6791/0] conn(0x7f0b1c133f50 0x7f0b1c134320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:09:18.463 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:18.463 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:18.472 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2026-02-12T04:09:17.771154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001671"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986308,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:21.474 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:21.659 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:21.659 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:21.668 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2026-02-12T04:09:19.771386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001606"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986308,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:21.670 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:21.847 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:21.848 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:21.857 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2026-02-12T04:09:21.771684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001630"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986309,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:21.857 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:21.857 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73278, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:12.334015+0000', 'last_change': '2026-02-12T04:09:12.334015+0000', 'last_active': '2026-02-12T04:09:12.334015+0000', 'last_peered': '2026-02-12T04:09:12.334015+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:12.334015+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:21.858 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:22.077 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:22.078 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":885,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:22.087 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:22.087 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:22.265 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:22.266 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:22.275 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2026-02-12T04:09:21.771684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001630"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986309,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:25.277 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:25.456 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:25.456 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:25.466 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":469,"stamp":"2026-02-12T04:09:23.771935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001634"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986309,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:25.466 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:25.643 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:25.644 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:25.653 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":469,"stamp":"2026-02-12T04:09:23.771935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001634"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986309,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:25.655 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:25.655 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73278, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:12.334015+0000', 'last_change': '2026-02-12T04:09:12.334015+0000', 'last_active': '2026-02-12T04:09:12.334015+0000', 'last_peered': '2026-02-12T04:09:12.334015+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:12.334015+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:25.656 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:25.876 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:25.877 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":889,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:25.887 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:25.887 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:26.064 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:26.064 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:26.073 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2026-02-12T04:09:25.772233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001633"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986310,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:29.075 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:29.255 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:29.256 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:29.264 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2026-02-12T04:09:27.772483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001617"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986310,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:29.265 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:29.443 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:29.444 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:29.453 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2026-02-12T04:09:27.772483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001617"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986310,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:29.454 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:29.454 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73278, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:12.334015+0000', 'last_change': '2026-02-12T04:09:12.334015+0000', 'last_active': '2026-02-12T04:09:12.334015+0000', 'last_peered': '2026-02-12T04:09:12.334015+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:12.334015+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:29.455 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:29.677 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:29.677 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":893,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:29.687 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:29.688 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:29.867 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:29.867 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:29.876 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2026-02-12T04:09:29.772776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001621"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986310,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:32.878 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:33.055 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:33.055 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:33.065 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2026-02-12T04:09:31.773010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001624"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986311,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:33.066 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:33.244 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:33.244 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:33.254 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2026-02-12T04:09:31.773010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001624"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986311,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:33.255 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:33.255 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73278, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:12.334015+0000', 'last_change': '2026-02-12T04:09:12.334015+0000', 'last_active': '2026-02-12T04:09:12.334015+0000', 'last_peered': '2026-02-12T04:09:12.334015+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:12.334015+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:33.255 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:33.473 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:33.473 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":896,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:33.483 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:33.483 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:33.663 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:33.664 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:33.673 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2026-02-12T04:09:31.773010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001624"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986311,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:36.675 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:36.854 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:36.854 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:36.863 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2026-02-12T04:09:35.773545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001609"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986312,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:36.864 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:37.046 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:37.047 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:37.057 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2026-02-12T04:09:35.773545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001609"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986312,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:37.058 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:37.059 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73278, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:12.334015+0000', 'last_change': '2026-02-12T04:09:12.334015+0000', 'last_active': '2026-02-12T04:09:12.334015+0000', 'last_peered': '2026-02-12T04:09:12.334015+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:12.334015+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:37.059 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:37.283 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:37.283 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":900,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:37.292 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:37.292 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:37.477 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:37.477 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:37.486 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2026-02-12T04:09:35.773545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001609"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986312,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:40.487 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:40.666 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:40.666 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:40.675 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2026-02-12T04:09:39.774089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001605"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986312,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:40.676 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:40.851 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:40.851 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:40.860 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2026-02-12T04:09:39.774089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001605"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986312,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:40.861 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:40.861 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73278, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:12.334015+0000', 'last_change': '2026-02-12T04:09:12.334015+0000', 'last_active': '2026-02-12T04:09:12.334015+0000', 'last_peered': '2026-02-12T04:09:12.334015+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:12.334015+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:40.861 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:41.081 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:41.081 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":904,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:41.091 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:41.091 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:41.270 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:41.270 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:41.279 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2026-02-12T04:09:39.774089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001605"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73278,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:12.334015+0000","last_change":"2026-02-12T04:09:12.334015+0000","last_active":"2026-02-12T04:09:12.334015+0000","last_peered":"2026-02-12T04:09:12.334015+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:12.334015+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986312,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:44.281 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:44.461 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:44.461 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:44.470 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2026-02-12T04:09:43.774504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986313,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:44.471 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:44.650 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:44.650 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:44.659 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2026-02-12T04:09:43.774504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986313,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:44.661 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:44.661 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73282, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:42.334615+0000', 'last_change': '2026-02-12T04:09:42.334615+0000', 'last_active': '2026-02-12T04:09:42.334615+0000', 'last_peered': '2026-02-12T04:09:42.334615+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:42.334615+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:44.661 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:44.904 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:44.904 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":908,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:44.914 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:44.915 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:45.095 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:45.096 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:45.104 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2026-02-12T04:09:43.774504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986313,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:48.106 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:48.285 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:48.286 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:48.295 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2026-02-12T04:09:47.775062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986314,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:48.296 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:48.473 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:48.473 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:48.482 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2026-02-12T04:09:47.775062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986314,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:48.483 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:48.483 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73282, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:42.334615+0000', 'last_change': '2026-02-12T04:09:42.334615+0000', 'last_active': '2026-02-12T04:09:42.334615+0000', 'last_peered': '2026-02-12T04:09:42.334615+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:42.334615+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:48.484 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:48.703 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:48.703 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":912,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:48.712 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:48.712 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:48.893 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:48.894 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:48.904 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2026-02-12T04:09:47.775062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986314,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:51.905 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:52.081 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:52.081 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:52.090 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2026-02-12T04:09:51.775614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001525"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986315,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:52.090 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:52.268 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:52.269 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:52.277 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2026-02-12T04:09:51.775614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001525"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986315,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:52.278 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:52.278 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73282, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:42.334615+0000', 'last_change': '2026-02-12T04:09:42.334615+0000', 'last_active': '2026-02-12T04:09:42.334615+0000', 'last_peered': '2026-02-12T04:09:42.334615+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:42.334615+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:52.279 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:52.506 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:52.506 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":916,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:52.515 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:52.516 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:52.693 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:52.694 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:52.702 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2026-02-12T04:09:51.775614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001525"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986315,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:55.703 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:55.883 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:55.884 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:55.893 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2026-02-12T04:09:55.776214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001710"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986316,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:55.894 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:56.073 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:56.074 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:56.082 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2026-02-12T04:09:55.776214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001710"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986316,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:56.083 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:09:56.084 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73282, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:42.334615+0000', 'last_change': '2026-02-12T04:09:42.334615+0000', 'last_active': '2026-02-12T04:09:42.334615+0000', 'last_peered': '2026-02-12T04:09:42.334615+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:42.334615+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:09:56.084 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:09:56.303 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:56.304 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":919,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:09:56.313 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:09:56.314 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:56.489 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:56.490 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:56.497 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2026-02-12T04:09:55.776214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001710"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986316,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:59.499 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:59.678 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:09:59.678 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:09:59.686 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":486,"stamp":"2026-02-12T04:09:57.776427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001652"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986316,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:09:59.687 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:09:59.958 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:09:59.957+0000 7fe84ffff640 0 --2- 10.20.193.103:0/4109698010 >> [v2:10.20.193.103:3300/0,v1:10.20.193.103:6789/0] conn(0x7fe858133f50 0x7fe85812b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:10:00.064 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:00.064 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:00.073 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2026-02-12T04:09:59.776655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986316,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:00.074 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:00.074 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73282, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:42.334615+0000', 'last_change': '2026-02-12T04:09:42.334615+0000', 'last_active': '2026-02-12T04:09:42.334615+0000', 'last_peered': '2026-02-12T04:09:42.334615+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:42.334615+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:00.074 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:00.303 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:00.303 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":923,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:10:00.312 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:00.313 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:00.488 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:00.489 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:00.497 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2026-02-12T04:09:59.776655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986316,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:03.499 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:03.678 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:03.678 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:03.687 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":488,"stamp":"2026-02-12T04:10:01.776917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001588"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986317,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:03.688 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:03.867 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:03.868 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:03.877 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2026-02-12T04:10:03.777178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001563"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986317,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:03.878 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:03.878 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73282, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:42.334615+0000', 'last_change': '2026-02-12T04:09:42.334615+0000', 'last_active': '2026-02-12T04:09:42.334615+0000', 'last_peered': '2026-02-12T04:09:42.334615+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:42.334615+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:03.879 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:04.097 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:04.098 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":927,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:10:04.107 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:04.107 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:04.282 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:04.283 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:04.291 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2026-02-12T04:10:03.777178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001563"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986317,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:07.293 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:07.478 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:07.478 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:07.487 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2026-02-12T04:10:05.777429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001516"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986318,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:07.488 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:07.662 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:07.663 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:07.671 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2026-02-12T04:10:05.777429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001516"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986318,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:07.672 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:07.672 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73282, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:42.334615+0000', 'last_change': '2026-02-12T04:09:42.334615+0000', 'last_active': '2026-02-12T04:09:42.334615+0000', 'last_peered': '2026-02-12T04:09:42.334615+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:42.334615+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:07.673 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:07.894 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:07.894 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":931,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:10:07.905 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:07.905 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:08.086 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:08.086 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:08.095 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2026-02-12T04:10:07.777750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001535"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986318,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:11.098 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:11.277 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:11.278 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:11.286 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2026-02-12T04:10:09.777946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001519"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986318,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:11.288 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:11.464 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:11.464 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:11.472 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2026-02-12T04:10:09.777946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001519"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986318,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:11.474 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:11.474 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73282, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:09:42.334615+0000', 'last_change': '2026-02-12T04:09:42.334615+0000', 'last_active': '2026-02-12T04:09:42.334615+0000', 'last_peered': '2026-02-12T04:09:42.334615+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:09:42.334615+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:11.474 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:11.699 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:11.699 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":935,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:10:11.708 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:11.708 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:11.887 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:11.888 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:11.897 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2026-02-12T04:10:11.778222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001566"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73282,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:09:42.334615+0000","last_change":"2026-02-12T04:09:42.334615+0000","last_active":"2026-02-12T04:09:42.334615+0000","last_peered":"2026-02-12T04:09:42.334615+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:09:42.334615+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:14.898 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:15.078 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:15.079 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:15.087 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2026-02-12T04:10:13.778444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001527"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:15.088 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:15.265 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:15.265 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:15.274 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2026-02-12T04:10:13.778444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001527"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:15.275 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:15.275 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73286, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:12.335213+0000', 'last_change': '2026-02-12T04:10:12.335213+0000', 'last_active': '2026-02-12T04:10:12.335213+0000', 'last_peered': '2026-02-12T04:10:12.335213+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:12.335213+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:15.276 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:15.498 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:15.498 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":938,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:10:15.509 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:15.509 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:15.687 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:15.688 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:15.696 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2026-02-12T04:10:13.778444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001527"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:18.698 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:18.877 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:18.877 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:18.886 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2026-02-12T04:10:17.778943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:18.888 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:19.157 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:10:19.155+0000 7febc5d74640 0 --2- 10.20.193.103:0/3356723717 >> [v2:10.20.193.103:3301/0,v1:10.20.193.103:6790/0] conn(0x7febc8057500 0x7febc8057950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:10:19.265 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:19.265 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:19.274 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2026-02-12T04:10:17.778943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:19.274 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:19.274 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73286, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:12.335213+0000', 'last_change': '2026-02-12T04:10:12.335213+0000', 'last_active': '2026-02-12T04:10:12.335213+0000', 'last_peered': '2026-02-12T04:10:12.335213+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:12.335213+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:19.275 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:19.497 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:19.497 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":942,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:10:19.506 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:19.507 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:19.685 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:19.685 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:19.695 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2026-02-12T04:10:17.778943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:22.696 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:22.879 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:22.879 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:22.889 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2026-02-12T04:10:21.779537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001590"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:22.890 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:23.071 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:23.071 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:23.080 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2026-02-12T04:10:21.779537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001590"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:23.081 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:23.081 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73286, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:12.335213+0000', 'last_change': '2026-02-12T04:10:12.335213+0000', 'last_active': '2026-02-12T04:10:12.335213+0000', 'last_peered': '2026-02-12T04:10:12.335213+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:12.335213+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:23.082 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:23.301 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:23.301 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":946,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:10:23.310 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:23.311 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:23.490 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:23.491 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:23.500 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2026-02-12T04:10:21.779537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001590"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:26.501 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:26.679 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:26.680 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:26.688 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2026-02-12T04:10:25.780060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001615"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986322,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:26.690 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:26.866 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:26.867 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:26.875 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2026-02-12T04:10:25.780060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001615"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986322,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:26.876 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:26.876 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73286, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:12.335213+0000', 'last_change': '2026-02-12T04:10:12.335213+0000', 'last_active': '2026-02-12T04:10:12.335213+0000', 'last_peered': '2026-02-12T04:10:12.335213+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:12.335213+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:26.876 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:27.098 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:27.098 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":950,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-12T04:07:23.756117+0000","services":{}},"progress_events":{}} 2026-02-12T04:10:27.108 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:27.108 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:27.290 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:27.291 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:27.300 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2026-02-12T04:10:25.780060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001615"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986322,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:30.301 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:30.484 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:30.484 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:30.493 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2026-02-12T04:10:29.780540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001596"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986322,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:30.494 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:30.673 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:30.673 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:30.682 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2026-02-12T04:10:29.780540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001596"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986322,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:30.683 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:30.683 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73286, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:12.335213+0000', 'last_change': '2026-02-12T04:10:12.335213+0000', 'last_active': '2026-02-12T04:10:12.335213+0000', 'last_peered': '2026-02-12T04:10:12.335213+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:12.335213+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:30.683 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:30.904 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:30.904 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":954,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":15,"modified":"2026-02-12T04:10:27.780355+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:10:30.914 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:30.915 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:31.095 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:31.095 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:31.104 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2026-02-12T04:10:29.780540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001596"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986322,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:34.106 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:34.289 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:34.289 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:34.298 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2026-02-12T04:10:33.781067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986323,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:34.300 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:34.477 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:34.478 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:34.486 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2026-02-12T04:10:33.781067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986323,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:34.488 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:34.488 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73286, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:12.335213+0000', 'last_change': '2026-02-12T04:10:12.335213+0000', 'last_active': '2026-02-12T04:10:12.335213+0000', 'last_peered': '2026-02-12T04:10:12.335213+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:12.335213+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:34.488 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:34.711 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:34.711 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":958,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":15,"modified":"2026-02-12T04:10:27.780355+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:10:34.720 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:34.721 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:34.899 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:34.899 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:34.908 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2026-02-12T04:10:33.781067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986323,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:37.910 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:38.089 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:38.090 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:38.098 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":506,"stamp":"2026-02-12T04:10:37.781586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001526"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986324,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:38.099 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:38.277 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:38.277 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:38.286 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":506,"stamp":"2026-02-12T04:10:37.781586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001526"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986324,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:38.287 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:38.287 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73286, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:12.335213+0000', 'last_change': '2026-02-12T04:10:12.335213+0000', 'last_active': '2026-02-12T04:10:12.335213+0000', 'last_peered': '2026-02-12T04:10:12.335213+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:12.335213+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:38.287 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:38.511 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:38.511 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":962,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":15,"modified":"2026-02-12T04:10:27.780355+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:10:38.520 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:38.521 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:38.699 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:38.699 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:38.708 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":506,"stamp":"2026-02-12T04:10:37.781586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001526"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:06:40.246963+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:06:40.246963+0000","last_peered":"2026-02-12T04:06:40.246963+0000","last_clean":"2026-02-12T04:06:40.246963+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:06:40.246963+0000","last_undegraded":"2026-02-12T04:06:40.246963+0000","last_fullsized":"2026-02-12T04:06:40.246963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986324,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:41.710 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:41.888 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:41.889 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:41.897 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2026-02-12T04:10:41.782143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:41.898 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:42.074 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:42.075 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:42.083 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2026-02-12T04:10:41.782143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:42.084 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:42.084 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73286, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:12.335213+0000', 'last_change': '2026-02-12T04:10:12.335213+0000', 'last_active': '2026-02-12T04:10:12.335213+0000', 'last_peered': '2026-02-12T04:10:12.335213+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:12.335213+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:42.085 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:42.313 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:42.314 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":965,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":15,"modified":"2026-02-12T04:10:27.780355+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:10:42.323 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:42.323 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:42.502 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:42.503 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:42.512 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2026-02-12T04:10:41.782143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73286,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:12.335213+0000","last_change":"2026-02-12T04:10:12.335213+0000","last_active":"2026-02-12T04:10:12.335213+0000","last_peered":"2026-02-12T04:10:12.335213+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:12.335213+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:45.513 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:45.692 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:45.692 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:45.701 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2026-02-12T04:10:43.782374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73290,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335583+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335583+0000","last_peered":"2026-02-12T04:10:42.335583+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335583+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:45.703 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:45.879 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:45.880 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:45.889 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2026-02-12T04:10:45.782683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001615"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73290,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335583+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335583+0000","last_peered":"2026-02-12T04:10:42.335583+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335583+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986326,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:45.890 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:45.890 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73290, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:42.335583+0000', 'last_change': '2026-02-12T04:10:42.335583+0000', 'last_active': '2026-02-12T04:10:42.335583+0000', 'last_peered': '2026-02-12T04:10:42.335583+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:42.335583+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:45.891 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:46.109 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:46.109 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":969,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-12T04:10:43.782428+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:10:46.119 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:46.119 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:46.300 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:46.300 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:46.310 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2026-02-12T04:10:45.782683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001615"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73290,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335583+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335583+0000","last_peered":"2026-02-12T04:10:42.335583+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335583+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986326,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:49.312 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:49.499 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:49.499 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:49.508 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2026-02-12T04:10:47.782946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001626"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73290,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335583+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335583+0000","last_peered":"2026-02-12T04:10:42.335583+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335583+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986326,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:49.509 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:49.688 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:49.689 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:49.698 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2026-02-12T04:10:47.782946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001626"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73290,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335583+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335583+0000","last_peered":"2026-02-12T04:10:42.335583+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335583+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986326,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:49.699 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:49.699 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73290, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:42.335583+0000', 'last_change': '2026-02-12T04:10:42.335583+0000', 'last_active': '2026-02-12T04:10:42.335583+0000', 'last_peered': '2026-02-12T04:10:42.335583+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:42.335583+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:49.700 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:49.921 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:49.922 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":973,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-12T04:10:43.782428+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:10:49.931 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:49.931 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:50.108 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:50.109 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:50.117 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2026-02-12T04:10:49.783243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001657"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73290,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335583+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335583+0000","last_peered":"2026-02-12T04:10:42.335583+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335583+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986326,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:53.119 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:53.301 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:53.301 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:53.310 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2026-02-12T04:10:51.783504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001634"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73290,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335583+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335583+0000","last_peered":"2026-02-12T04:10:42.335583+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335583+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986327,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:53.311 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:53.490 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:53.491 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:53.499 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2026-02-12T04:10:51.783504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001634"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73290,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335583+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335583+0000","last_peered":"2026-02-12T04:10:42.335583+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335583+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986327,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:53.500 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:53.501 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73290, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:42.335583+0000', 'last_change': '2026-02-12T04:10:42.335583+0000', 'last_active': '2026-02-12T04:10:42.335583+0000', 'last_peered': '2026-02-12T04:10:42.335583+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:42.335583+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:53.501 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:53.721 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:53.721 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":977,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-12T04:10:43.782428+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:10:53.730 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:53.730 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:53.908 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:53.908 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:53.917 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2026-02-12T04:10:53.783830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001687"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986327,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:56.919 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:57.098 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:57.099 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:57.107 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2026-02-12T04:10:55.784107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001733"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986328,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:57.108 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:57.289 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:57.290 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:57.298 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2026-02-12T04:10:55.784107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001733"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986328,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:10:57.300 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:10:57.300 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73291, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:42.335608+0000', 'last_change': '2026-02-12T04:10:42.335583+0000', 'last_active': '2026-02-12T04:10:42.335608+0000', 'last_peered': '2026-02-12T04:10:42.335608+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:42.335608+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:10:57.300 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:10:57.519 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:57.519 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":981,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-12T04:10:43.782428+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:10:57.529 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:10:57.529 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:10:57.709 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:10:57.709 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:10:57.719 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2026-02-12T04:10:55.784107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001733"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986328,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:00.722 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:00.899 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:00.899 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:00.908 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2026-02-12T04:10:59.784674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001727"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986328,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:00.909 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:01.084 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:01.084 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:01.093 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2026-02-12T04:10:59.784674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001727"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986328,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:01.094 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:01.094 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73291, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:42.335608+0000', 'last_change': '2026-02-12T04:10:42.335583+0000', 'last_active': '2026-02-12T04:10:42.335608+0000', 'last_peered': '2026-02-12T04:10:42.335608+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:42.335608+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:01.094 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:01.323 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:01.323 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":984,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:01.331 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:01.332 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:01.506 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:01.507 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:01.515 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2026-02-12T04:10:59.784674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001727"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986328,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:04.517 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:04.692 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:04.693 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:04.701 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":519,"stamp":"2026-02-12T04:11:03.785262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001758"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986329,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:04.702 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:04.878 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:04.879 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:04.887 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":519,"stamp":"2026-02-12T04:11:03.785262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001758"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986329,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:04.889 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:04.889 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73291, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:42.335608+0000', 'last_change': '2026-02-12T04:10:42.335583+0000', 'last_active': '2026-02-12T04:10:42.335608+0000', 'last_peered': '2026-02-12T04:10:42.335608+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:42.335608+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:04.889 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:05.108 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:05.108 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":988,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:05.117 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:05.117 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:05.386 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:11:05.385+0000 7f9bdf577640 0 --2- 10.20.193.103:0/1646057668 >> [v2:10.20.193.103:3300/0,v1:10.20.193.103:6789/0] conn(0x7f9be012e980 0x7f9be0183c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:11:05.498 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:05.498 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:05.507 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":519,"stamp":"2026-02-12T04:11:03.785262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001758"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986329,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:08.509 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:08.687 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:08.688 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:08.696 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":521,"stamp":"2026-02-12T04:11:07.785799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001692"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986330,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:08.697 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:08.875 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:08.876 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:08.885 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":521,"stamp":"2026-02-12T04:11:07.785799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001692"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986330,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:08.886 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:08.886 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73291, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:42.335608+0000', 'last_change': '2026-02-12T04:10:42.335583+0000', 'last_active': '2026-02-12T04:10:42.335608+0000', 'last_peered': '2026-02-12T04:10:42.335608+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:42.335608+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:08.887 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:09.110 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:09.110 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":992,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:09.119 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:09.120 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:09.298 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:09.298 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:09.307 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":521,"stamp":"2026-02-12T04:11:07.785799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001692"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986330,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:12.309 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:12.487 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:12.487 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:12.496 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2026-02-12T04:11:11.786265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001591"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:12.497 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:12.675 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:12.676 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:12.684 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2026-02-12T04:11:11.786265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001591"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:12.685 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:12.685 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73291, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:10:42.335608+0000', 'last_change': '2026-02-12T04:10:42.335583+0000', 'last_active': '2026-02-12T04:10:42.335608+0000', 'last_peered': '2026-02-12T04:10:42.335608+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:10:42.335608+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:12.686 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:12.909 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:12.909 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":996,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:12.919 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:12.920 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:13.189 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:11:13.187+0000 7fd511186640 0 --2- 10.20.193.103:0/3295252450 >> [v2:10.20.193.103:3302/0,v1:10.20.193.103:6791/0] conn(0x7fd50c11a190 0x7fd50c11a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:11:13.299 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:13.300 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:13.309 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2026-02-12T04:11:11.786265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001591"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73291,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:10:42.335608+0000","last_change":"2026-02-12T04:10:42.335583+0000","last_active":"2026-02-12T04:10:42.335608+0000","last_peered":"2026-02-12T04:10:42.335608+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:10:42.335608+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:16.311 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:16.489 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:16.489 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:16.498 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2026-02-12T04:11:15.786875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001612"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:16.499 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:16.676 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:16.677 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:16.685 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2026-02-12T04:11:15.786875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001612"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:16.686 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:16.686 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73295, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:12.335960+0000', 'last_change': '2026-02-12T04:11:12.335960+0000', 'last_active': '2026-02-12T04:11:12.335960+0000', 'last_peered': '2026-02-12T04:11:12.335960+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:12.335960+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:16.687 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:16.909 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:16.910 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1000,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:16.919 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:16.919 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:17.099 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:17.100 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:17.109 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2026-02-12T04:11:15.786875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001612"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:20.110 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:20.289 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:20.289 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:20.297 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":527,"stamp":"2026-02-12T04:11:19.787475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001675"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:20.298 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:20.476 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:20.476 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:20.485 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":527,"stamp":"2026-02-12T04:11:19.787475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001675"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:20.486 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:20.486 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73295, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:12.335960+0000', 'last_change': '2026-02-12T04:11:12.335960+0000', 'last_active': '2026-02-12T04:11:12.335960+0000', 'last_peered': '2026-02-12T04:11:12.335960+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:12.335960+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:20.486 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:20.706 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:20.706 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1004,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:20.715 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:20.716 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:20.895 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:20.895 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:20.904 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":527,"stamp":"2026-02-12T04:11:19.787475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001675"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:23.906 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:24.089 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:24.089 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:24.098 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":529,"stamp":"2026-02-12T04:11:23.787885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001620"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:24.099 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:24.278 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:24.279 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:24.287 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":529,"stamp":"2026-02-12T04:11:23.787885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001620"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:24.288 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:24.288 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73295, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:12.335960+0000', 'last_change': '2026-02-12T04:11:12.335960+0000', 'last_active': '2026-02-12T04:11:12.335960+0000', 'last_peered': '2026-02-12T04:11:12.335960+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:12.335960+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:24.289 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:24.516 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:24.516 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1008,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:24.526 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:24.526 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:24.704 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:24.705 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:24.714 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":529,"stamp":"2026-02-12T04:11:23.787885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001620"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:27.715 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:27.894 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:27.894 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:27.903 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2026-02-12T04:11:27.788425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001550"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986334,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:27.904 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:28.080 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:28.081 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:28.089 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2026-02-12T04:11:27.788425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001550"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986334,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:28.090 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:28.090 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73295, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:12.335960+0000', 'last_change': '2026-02-12T04:11:12.335960+0000', 'last_active': '2026-02-12T04:11:12.335960+0000', 'last_peered': '2026-02-12T04:11:12.335960+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:12.335960+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:28.091 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:28.311 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:28.312 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1011,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:28.321 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:28.321 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:28.500 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:28.500 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:28.510 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2026-02-12T04:11:27.788425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001550"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986334,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:31.512 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:31.694 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:31.694 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:31.704 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":532,"stamp":"2026-02-12T04:11:29.788631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001452"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986334,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:31.705 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:31.889 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:31.889 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:31.898 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2026-02-12T04:11:31.788934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001458"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:31.899 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:31.899 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73295, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:12.335960+0000', 'last_change': '2026-02-12T04:11:12.335960+0000', 'last_active': '2026-02-12T04:11:12.335960+0000', 'last_peered': '2026-02-12T04:11:12.335960+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:12.335960+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:31.899 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:32.119 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:32.119 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1015,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:32.128 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:32.128 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:32.306 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:32.307 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:32.316 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2026-02-12T04:11:31.788934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001458"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:35.318 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:35.496 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:35.496 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:35.505 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":534,"stamp":"2026-02-12T04:11:33.789164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001489"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:35.507 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:35.688 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:35.688 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:35.697 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":534,"stamp":"2026-02-12T04:11:33.789164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001489"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:35.698 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:35.698 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73295, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:12.335960+0000', 'last_change': '2026-02-12T04:11:12.335960+0000', 'last_active': '2026-02-12T04:11:12.335960+0000', 'last_peered': '2026-02-12T04:11:12.335960+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:12.335960+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:35.698 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:35.919 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:35.919 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1019,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:35.928 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:35.928 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:36.106 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:36.106 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:36.115 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2026-02-12T04:11:35.789443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001557"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:39.116 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:39.301 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:39.301 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:39.310 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":536,"stamp":"2026-02-12T04:11:37.789686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001573"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:39.311 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:39.488 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:39.489 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:39.498 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":536,"stamp":"2026-02-12T04:11:37.789686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001573"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:39.499 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:39.499 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73295, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:12.335960+0000', 'last_change': '2026-02-12T04:11:12.335960+0000', 'last_active': '2026-02-12T04:11:12.335960+0000', 'last_peered': '2026-02-12T04:11:12.335960+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:12.335960+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:39.500 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:39.726 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:39.726 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1023,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:39.735 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:39.735 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:39.909 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:39.910 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:39.918 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":537,"stamp":"2026-02-12T04:11:39.789981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001556"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:42.920 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:43.101 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:43.101 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:43.111 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2026-02-12T04:11:41.790259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001628"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:43.112 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:43.291 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:43.292 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:43.300 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2026-02-12T04:11:41.790259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001628"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:43.301 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:43.302 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73295, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:12.335960+0000', 'last_change': '2026-02-12T04:11:12.335960+0000', 'last_active': '2026-02-12T04:11:12.335960+0000', 'last_peered': '2026-02-12T04:11:12.335960+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:12.335960+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:43.302 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:43.518 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:43.518 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1027,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:43.529 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:43.529 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:43.710 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:43.711 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:43.718 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2026-02-12T04:11:41.790259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001628"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73295,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:12.335960+0000","last_change":"2026-02-12T04:11:12.335960+0000","last_active":"2026-02-12T04:11:12.335960+0000","last_peered":"2026-02-12T04:11:12.335960+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:12.335960+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:46.721 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:46.901 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:46.902 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:46.910 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":540,"stamp":"2026-02-12T04:11:45.790732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001568"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986338,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:46.911 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:47.086 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:47.087 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:47.095 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":540,"stamp":"2026-02-12T04:11:45.790732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001568"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986338,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:47.096 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:47.096 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73299, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:42.336463+0000', 'last_change': '2026-02-12T04:11:42.336463+0000', 'last_active': '2026-02-12T04:11:42.336463+0000', 'last_peered': '2026-02-12T04:11:42.336463+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:42.336463+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:47.097 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:47.326 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:47.327 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1030,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:47.336 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:47.336 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:47.513 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:47.514 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:47.523 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":540,"stamp":"2026-02-12T04:11:45.790732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001568"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986338,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:50.525 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:50.705 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:50.706 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:50.715 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":542,"stamp":"2026-02-12T04:11:49.791180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001494"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986338,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:50.716 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:50.895 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:50.895 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:50.904 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":542,"stamp":"2026-02-12T04:11:49.791180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001494"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986338,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:50.905 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:50.906 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73299, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:42.336463+0000', 'last_change': '2026-02-12T04:11:42.336463+0000', 'last_active': '2026-02-12T04:11:42.336463+0000', 'last_peered': '2026-02-12T04:11:42.336463+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:42.336463+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:50.906 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:51.129 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:51.129 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1034,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:51.140 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:51.140 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:51.319 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:51.320 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:51.329 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":542,"stamp":"2026-02-12T04:11:49.791180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001494"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986338,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:54.331 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:54.509 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:54.510 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:54.519 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":544,"stamp":"2026-02-12T04:11:53.791695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001435"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:54.520 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:54.700 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:54.700 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:54.709 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":544,"stamp":"2026-02-12T04:11:53.791695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001435"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:54.710 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:54.710 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73299, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:42.336463+0000', 'last_change': '2026-02-12T04:11:42.336463+0000', 'last_active': '2026-02-12T04:11:42.336463+0000', 'last_peered': '2026-02-12T04:11:42.336463+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:42.336463+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:54.710 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:54.933 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:54.934 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1038,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:54.943 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:54.944 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:55.121 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:55.121 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:55.130 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":544,"stamp":"2026-02-12T04:11:53.791695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001435"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:58.132 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:58.309 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:58.310 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:58.318 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":546,"stamp":"2026-02-12T04:11:57.792244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001511"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986340,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:58.319 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:58.498 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:58.498 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:58.508 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":546,"stamp":"2026-02-12T04:11:57.792244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001511"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986340,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:11:58.509 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:11:58.509 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73299, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:42.336463+0000', 'last_change': '2026-02-12T04:11:42.336463+0000', 'last_active': '2026-02-12T04:11:42.336463+0000', 'last_peered': '2026-02-12T04:11:42.336463+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:42.336463+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:11:58.509 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:11:58.734 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:58.734 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1042,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:11:58.743 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:11:58.744 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:11:58.920 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:11:58.921 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:11:58.930 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":546,"stamp":"2026-02-12T04:11:57.792244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001511"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986340,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:01.931 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:02.109 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:02.109 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:02.118 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":548,"stamp":"2026-02-12T04:12:01.792799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001618"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986341,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:02.119 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:02.302 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:02.302 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:02.311 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":548,"stamp":"2026-02-12T04:12:01.792799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001618"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986341,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:02.312 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:02.312 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73299, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:42.336463+0000', 'last_change': '2026-02-12T04:11:42.336463+0000', 'last_active': '2026-02-12T04:11:42.336463+0000', 'last_peered': '2026-02-12T04:11:42.336463+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:42.336463+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:02.313 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:02.536 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:02.537 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1046,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:02.547 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:02.547 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:02.726 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:02.726 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:02.734 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":548,"stamp":"2026-02-12T04:12:01.792799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001618"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986341,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:05.736 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:05.916 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:05.917 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:05.925 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":550,"stamp":"2026-02-12T04:12:05.793296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001601"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986342,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:05.926 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:06.103 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:06.104 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:06.113 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":550,"stamp":"2026-02-12T04:12:05.793296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001601"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986342,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:06.113 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:06.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73299, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:42.336463+0000', 'last_change': '2026-02-12T04:11:42.336463+0000', 'last_active': '2026-02-12T04:11:42.336463+0000', 'last_peered': '2026-02-12T04:11:42.336463+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:42.336463+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:06.114 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:06.338 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:06.339 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1049,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:06.348 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:06.349 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:06.529 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:06.529 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:06.539 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":550,"stamp":"2026-02-12T04:12:05.793296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001601"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986342,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:09.540 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:09.717 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:09.718 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:09.726 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2026-02-12T04:12:07.793529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001601"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986342,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:09.727 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:09.904 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:09.905 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:09.913 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":552,"stamp":"2026-02-12T04:12:09.793771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001526"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986342,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:09.914 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:09.914 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73299, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:42.336463+0000', 'last_change': '2026-02-12T04:11:42.336463+0000', 'last_active': '2026-02-12T04:11:42.336463+0000', 'last_peered': '2026-02-12T04:11:42.336463+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:42.336463+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:09.914 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:10.137 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:10.137 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1053,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:10.147 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:10.147 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:10.325 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:10.325 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:10.334 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":552,"stamp":"2026-02-12T04:12:09.793771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001526"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986342,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:13.335 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:13.515 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:13.516 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:13.525 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":553,"stamp":"2026-02-12T04:12:11.794061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001562"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986343,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:13.526 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:13.703 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:13.704 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:13.712 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":553,"stamp":"2026-02-12T04:12:11.794061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001562"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73299,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:11:42.336463+0000","last_change":"2026-02-12T04:11:42.336463+0000","last_active":"2026-02-12T04:11:42.336463+0000","last_peered":"2026-02-12T04:11:42.336463+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:11:42.336463+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986343,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:13.713 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:13.713 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73299, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:11:42.336463+0000', 'last_change': '2026-02-12T04:11:42.336463+0000', 'last_active': '2026-02-12T04:11:42.336463+0000', 'last_peered': '2026-02-12T04:11:42.336463+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:11:42.336463+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:13.714 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:13.937 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:13.937 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1057,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:13.946 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:13.947 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:14.124 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:14.125 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:14.133 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":554,"stamp":"2026-02-12T04:12:13.794364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001564"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986343,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:17.135 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:17.317 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:17.317 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:17.326 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":555,"stamp":"2026-02-12T04:12:15.794657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986344,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:17.327 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:17.507 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:17.507 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:17.517 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":555,"stamp":"2026-02-12T04:12:15.794657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986344,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:17.518 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:17.518 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73303, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:12.337012+0000', 'last_change': '2026-02-12T04:12:12.337012+0000', 'last_active': '2026-02-12T04:12:12.337012+0000', 'last_peered': '2026-02-12T04:12:12.337012+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:12.337012+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:17.518 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:17.742 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:17.742 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1061,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:17.753 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:17.753 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:17.932 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:17.933 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:17.942 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":556,"stamp":"2026-02-12T04:12:17.794967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001670"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986344,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:20.943 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:21.218 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:21.219 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:21.229 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":557,"stamp":"2026-02-12T04:12:19.795234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001705"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986344,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:21.230 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:21.408 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:21.408 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:21.417 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":557,"stamp":"2026-02-12T04:12:19.795234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001705"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986344,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:21.418 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:21.418 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73303, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:12.337012+0000', 'last_change': '2026-02-12T04:12:12.337012+0000', 'last_active': '2026-02-12T04:12:12.337012+0000', 'last_peered': '2026-02-12T04:12:12.337012+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:12.337012+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:21.418 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:21.640 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:21.641 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1065,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:21.650 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:21.650 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:21.828 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:21.828 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:21.837 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":558,"stamp":"2026-02-12T04:12:21.795524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001753"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986345,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:24.839 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:25.021 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:25.022 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:25.031 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":559,"stamp":"2026-02-12T04:12:23.795763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986345,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:25.032 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:25.210 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:25.211 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:25.219 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":559,"stamp":"2026-02-12T04:12:23.795763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986345,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:25.220 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:25.220 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73303, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:12.337012+0000', 'last_change': '2026-02-12T04:12:12.337012+0000', 'last_active': '2026-02-12T04:12:12.337012+0000', 'last_peered': '2026-02-12T04:12:12.337012+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:12.337012+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:25.221 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:25.443 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:25.444 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1068,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-12T04:10:59.784709+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:25.453 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:25.453 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:25.728 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:25.728 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:25.740 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":559,"stamp":"2026-02-12T04:12:23.795763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986345,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:28.741 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:28.922 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:28.923 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:28.932 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":561,"stamp":"2026-02-12T04:12:27.796237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001579"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986346,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:28.932 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:29.110 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:29.111 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:29.120 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":561,"stamp":"2026-02-12T04:12:27.796237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001579"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986346,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:29.120 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:29.120 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73303, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:12.337012+0000', 'last_change': '2026-02-12T04:12:12.337012+0000', 'last_active': '2026-02-12T04:12:12.337012+0000', 'last_peered': '2026-02-12T04:12:12.337012+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:12.337012+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:29.120 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:29.341 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:29.341 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1072,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:29.350 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:29.350 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:29.525 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:29.525 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:29.535 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":561,"stamp":"2026-02-12T04:12:27.796237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001579"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986346,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:32.536 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:32.719 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:32.719 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:32.728 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":563,"stamp":"2026-02-12T04:12:31.796776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986347,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:32.728 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:32.906 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:32.906 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:32.915 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":563,"stamp":"2026-02-12T04:12:31.796776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986347,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:32.915 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:32.916 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73303, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:12.337012+0000', 'last_change': '2026-02-12T04:12:12.337012+0000', 'last_active': '2026-02-12T04:12:12.337012+0000', 'last_peered': '2026-02-12T04:12:12.337012+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:12.337012+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:32.916 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:33.132 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:33.133 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1076,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:33.142 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:33.142 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:33.324 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:33.324 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:33.333 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":563,"stamp":"2026-02-12T04:12:31.796776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986347,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:36.334 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:36.521 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:36.521 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:36.530 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":565,"stamp":"2026-02-12T04:12:35.797313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001550"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986348,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:36.531 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:36.708 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:36.709 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:36.718 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":565,"stamp":"2026-02-12T04:12:35.797313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001550"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986348,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:36.719 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:36.719 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73303, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:12.337012+0000', 'last_change': '2026-02-12T04:12:12.337012+0000', 'last_active': '2026-02-12T04:12:12.337012+0000', 'last_peered': '2026-02-12T04:12:12.337012+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:12.337012+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:36.720 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:36.991 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:12:36.991+0000 7fbe2b7fe640 0 --2- 10.20.193.103:0/4057720538 >> [v2:10.20.193.103:3302/0,v1:10.20.193.103:6791/0] conn(0x7fbe2c12afd0 0x7fbe2c143f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:12:37.144 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:37.144 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1080,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:37.154 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:37.154 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:37.333 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:37.333 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:37.342 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":565,"stamp":"2026-02-12T04:12:35.797313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001550"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986348,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:40.343 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:40.521 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:40.521 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:40.530 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":567,"stamp":"2026-02-12T04:12:39.797886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001649"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986348,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:40.530 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:40.713 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:40.713 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:40.722 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":567,"stamp":"2026-02-12T04:12:39.797886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001649"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986348,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:40.723 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:40.723 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73303, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:12.337012+0000', 'last_change': '2026-02-12T04:12:12.337012+0000', 'last_active': '2026-02-12T04:12:12.337012+0000', 'last_peered': '2026-02-12T04:12:12.337012+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:12.337012+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:40.723 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:40.942 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:40.943 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1084,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:40.952 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:40.953 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:41.127 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:41.127 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:41.135 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":567,"stamp":"2026-02-12T04:12:39.797886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001649"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73303,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:12.337012+0000","last_change":"2026-02-12T04:12:12.337012+0000","last_active":"2026-02-12T04:12:12.337012+0000","last_peered":"2026-02-12T04:12:12.337012+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:12.337012+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:08:40.257262+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:08:40.257262+0000","last_peered":"2026-02-12T04:08:40.257262+0000","last_clean":"2026-02-12T04:08:40.257262+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:08:40.257262+0000","last_undegraded":"2026-02-12T04:08:40.257262+0000","last_fullsized":"2026-02-12T04:08:40.257262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986348,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:44.136 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:44.315 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:44.315 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:44.324 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":569,"stamp":"2026-02-12T04:12:43.798407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001631"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73307,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337354+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337354+0000","last_peered":"2026-02-12T04:12:42.337354+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337354+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986349,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:44.324 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:44.504 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:44.504 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:44.514 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":569,"stamp":"2026-02-12T04:12:43.798407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001631"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73307,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337354+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337354+0000","last_peered":"2026-02-12T04:12:42.337354+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337354+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986349,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:44.514 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:44.514 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73307, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:42.337354+0000', 'last_change': '2026-02-12T04:12:42.337354+0000', 'last_active': '2026-02-12T04:12:42.337354+0000', 'last_peered': '2026-02-12T04:12:42.337354+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:42.337354+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:44.515 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:44.733 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:44.733 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1088,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:44.742 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:44.742 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:44.918 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:44.918 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:44.927 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":569,"stamp":"2026-02-12T04:12:43.798407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001631"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73307,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337354+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337354+0000","last_peered":"2026-02-12T04:12:42.337354+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337354+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986349,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:47.928 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:48.205 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:48.206 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:48.217 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":571,"stamp":"2026-02-12T04:12:47.798923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001609"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73307,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337354+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337354+0000","last_peered":"2026-02-12T04:12:42.337354+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337354+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986350,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:48.218 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:48.400 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:48.401 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:48.409 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":571,"stamp":"2026-02-12T04:12:47.798923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001609"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73307,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337354+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337354+0000","last_peered":"2026-02-12T04:12:42.337354+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337354+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986350,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:48.410 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:48.410 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73307, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:42.337354+0000', 'last_change': '2026-02-12T04:12:42.337354+0000', 'last_active': '2026-02-12T04:12:42.337354+0000', 'last_peered': '2026-02-12T04:12:42.337354+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:42.337354+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:48.410 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:48.632 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:48.632 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1092,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:48.641 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:48.642 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:48.822 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:48.823 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:48.830 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":571,"stamp":"2026-02-12T04:12:47.798923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001609"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73307,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337354+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337354+0000","last_peered":"2026-02-12T04:12:42.337354+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337354+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986350,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:51.833 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:52.011 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:52.011 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:52.019 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":573,"stamp":"2026-02-12T04:12:51.799558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001671"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73307,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337354+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337354+0000","last_peered":"2026-02-12T04:12:42.337354+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337354+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986351,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:52.021 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:52.198 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:52.199 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:52.208 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":573,"stamp":"2026-02-12T04:12:51.799558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001671"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73307,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337354+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337354+0000","last_peered":"2026-02-12T04:12:42.337354+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337354+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986351,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:52.209 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:52.209 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73307, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:42.337354+0000', 'last_change': '2026-02-12T04:12:42.337354+0000', 'last_active': '2026-02-12T04:12:42.337354+0000', 'last_peered': '2026-02-12T04:12:42.337354+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:42.337354+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:52.210 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:52.432 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:52.433 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1095,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:52.442 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:52.442 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:52.621 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:52.622 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:52.630 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":573,"stamp":"2026-02-12T04:12:51.799558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001671"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73307,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337354+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337354+0000","last_peered":"2026-02-12T04:12:42.337354+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337354+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986351,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:55.632 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:55.812 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:55.813 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:55.821 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":575,"stamp":"2026-02-12T04:12:55.800241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001833"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986352,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:55.823 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:56.098 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:56.098 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:56.107 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":575,"stamp":"2026-02-12T04:12:55.800241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001833"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986352,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:56.108 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:56.109 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73308, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:42.337385+0000', 'last_change': '2026-02-12T04:12:42.337354+0000', 'last_active': '2026-02-12T04:12:42.337385+0000', 'last_peered': '2026-02-12T04:12:42.337385+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:42.337385+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:56.109 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:12:56.335 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:56.335 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1099,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:12:56.344 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:12:56.344 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:56.521 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:56.521 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:56.530 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":575,"stamp":"2026-02-12T04:12:55.800241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001833"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986352,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:59.532 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:59.712 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:59.713 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:59.722 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":576,"stamp":"2026-02-12T04:12:57.800500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001870"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986352,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:59.723 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:12:59.902 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:12:59.903 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:12:59.912 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":577,"stamp":"2026-02-12T04:12:59.800787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001864"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986352,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:12:59.913 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:12:59.913 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73308, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:42.337385+0000', 'last_change': '2026-02-12T04:12:42.337354+0000', 'last_active': '2026-02-12T04:12:42.337385+0000', 'last_peered': '2026-02-12T04:12:42.337385+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:42.337385+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:12:59.914 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:00.134 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:00.134 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1103,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:00.143 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:00.144 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:00.323 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:00.324 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:00.333 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":577,"stamp":"2026-02-12T04:12:59.800787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001864"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986352,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:03.334 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:03.511 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:03.511 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:03.520 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":578,"stamp":"2026-02-12T04:13:01.801050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001834"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986353,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:03.520 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:03.695 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:03.695 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:03.703 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":578,"stamp":"2026-02-12T04:13:01.801050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001834"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986353,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:03.704 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:03.704 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73308, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:42.337385+0000', 'last_change': '2026-02-12T04:12:42.337354+0000', 'last_active': '2026-02-12T04:12:42.337385+0000', 'last_peered': '2026-02-12T04:12:42.337385+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:42.337385+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:03.704 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:03.926 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:03.926 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1107,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:03.936 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:03.936 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:04.111 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:04.111 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:04.120 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":579,"stamp":"2026-02-12T04:13:03.801358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001800"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986353,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:07.121 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:07.302 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:07.302 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:07.311 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":580,"stamp":"2026-02-12T04:13:05.801621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001716"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:07.312 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:07.585 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:07.585 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:07.595 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":580,"stamp":"2026-02-12T04:13:05.801621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001716"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:07.596 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:07.596 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73308, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:42.337385+0000', 'last_change': '2026-02-12T04:12:42.337354+0000', 'last_active': '2026-02-12T04:12:42.337385+0000', 'last_peered': '2026-02-12T04:12:42.337385+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:42.337385+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:07.596 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:07.818 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:07.818 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1111,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:07.828 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:07.828 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:08.005 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:08.005 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:08.015 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":581,"stamp":"2026-02-12T04:13:07.801936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001695"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:11.017 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:11.206 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:11.206 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:11.216 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":582,"stamp":"2026-02-12T04:13:09.802189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001688"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:11.217 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:11.393 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:11.393 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:11.402 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":582,"stamp":"2026-02-12T04:13:09.802189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001688"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:11.403 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:11.403 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73308, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:12:42.337385+0000', 'last_change': '2026-02-12T04:12:42.337354+0000', 'last_active': '2026-02-12T04:12:42.337385+0000', 'last_peered': '2026-02-12T04:12:42.337385+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:12:42.337385+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:11.403 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:11.625 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:11.625 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1115,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:11.635 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:11.635 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:11.811 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:11.811 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:11.820 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":583,"stamp":"2026-02-12T04:13:11.802491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001704"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73308,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:12:42.337385+0000","last_change":"2026-02-12T04:12:42.337354+0000","last_active":"2026-02-12T04:12:42.337385+0000","last_peered":"2026-02-12T04:12:42.337385+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:12:42.337385+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986355,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:14.822 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:15.000 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:15.000 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:15.009 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":584,"stamp":"2026-02-12T04:13:13.802760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001710"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986355,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:15.010 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:15.190 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:15.191 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:15.200 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":584,"stamp":"2026-02-12T04:13:13.802760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001710"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986355,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:15.201 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:15.201 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73312, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:12.337906+0000', 'last_change': '2026-02-12T04:13:12.337906+0000', 'last_active': '2026-02-12T04:13:12.337906+0000', 'last_peered': '2026-02-12T04:13:12.337906+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:12.337906+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:15.202 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:15.423 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:15.423 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1118,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:15.432 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:15.432 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:15.611 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:15.612 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:15.621 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":584,"stamp":"2026-02-12T04:13:13.802760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001710"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986355,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:18.623 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:18.802 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:18.802 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:18.811 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":586,"stamp":"2026-02-12T04:13:17.803210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:18.812 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:18.990 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:18.991 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:19.000 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":586,"stamp":"2026-02-12T04:13:17.803210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:19.001 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:19.001 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73312, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:12.337906+0000', 'last_change': '2026-02-12T04:13:12.337906+0000', 'last_active': '2026-02-12T04:13:12.337906+0000', 'last_peered': '2026-02-12T04:13:12.337906+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:12.337906+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:19.001 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:19.225 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:19.225 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1122,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:19.234 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:19.235 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:19.410 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:19.410 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:19.419 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":586,"stamp":"2026-02-12T04:13:17.803210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:22.421 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:22.599 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:22.600 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:22.608 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":588,"stamp":"2026-02-12T04:13:21.803708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001519"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986357,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:22.609 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:22.879 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:13:22.878+0000 7f200e575640 0 --2- 10.20.193.103:0/1536330658 >> [v2:10.20.193.103:3302/0,v1:10.20.193.103:6791/0] conn(0x7f201011ab00 0x7f201012ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:13:22.986 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:22.987 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:22.996 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":588,"stamp":"2026-02-12T04:13:21.803708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001519"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986357,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:22.997 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:22.997 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73312, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:12.337906+0000', 'last_change': '2026-02-12T04:13:12.337906+0000', 'last_active': '2026-02-12T04:13:12.337906+0000', 'last_peered': '2026-02-12T04:13:12.337906+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:12.337906+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:22.997 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:23.218 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:23.218 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1126,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:23.227 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:23.228 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:23.409 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:23.409 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:23.419 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":588,"stamp":"2026-02-12T04:13:21.803708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001519"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986357,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:26.421 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:26.607 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:26.607 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:26.616 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":590,"stamp":"2026-02-12T04:13:25.804299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001538"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:26.617 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:26.890 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:26.891 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:26.899 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":590,"stamp":"2026-02-12T04:13:25.804299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001538"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:26.900 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:26.901 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73312, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:12.337906+0000', 'last_change': '2026-02-12T04:13:12.337906+0000', 'last_active': '2026-02-12T04:13:12.337906+0000', 'last_peered': '2026-02-12T04:13:12.337906+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:12.337906+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:26.901 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:27.128 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:27.128 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1130,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:27.138 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:27.138 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:27.317 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:27.317 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:27.326 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":590,"stamp":"2026-02-12T04:13:25.804299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001538"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:30.328 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:30.506 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:30.506 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:30.516 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":592,"stamp":"2026-02-12T04:13:29.804882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001672"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:30.517 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:30.693 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:30.694 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:30.702 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":592,"stamp":"2026-02-12T04:13:29.804882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001672"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:30.703 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:30.704 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73312, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:12.337906+0000', 'last_change': '2026-02-12T04:13:12.337906+0000', 'last_active': '2026-02-12T04:13:12.337906+0000', 'last_peered': '2026-02-12T04:13:12.337906+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:12.337906+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:30.704 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:30.928 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:30.929 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1134,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:30.939 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:30.939 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:31.120 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:31.120 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:31.130 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":592,"stamp":"2026-02-12T04:13:29.804882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001672"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:34.131 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:34.309 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:34.310 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:34.318 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":594,"stamp":"2026-02-12T04:13:33.805419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001710"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986359,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:34.319 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:34.499 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:34.499 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:34.508 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":594,"stamp":"2026-02-12T04:13:33.805419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001710"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986359,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:34.509 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:34.509 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73312, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:12.337906+0000', 'last_change': '2026-02-12T04:13:12.337906+0000', 'last_active': '2026-02-12T04:13:12.337906+0000', 'last_peered': '2026-02-12T04:13:12.337906+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:12.337906+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:34.510 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:34.729 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:34.729 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1138,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:34.739 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:34.740 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:34.916 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:34.917 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:34.925 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":594,"stamp":"2026-02-12T04:13:33.805419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001710"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986359,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:37.926 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:38.109 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:38.109 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:38.118 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2026-02-12T04:13:37.805970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001671"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986360,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:38.119 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:38.296 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:38.296 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:38.305 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2026-02-12T04:13:37.805970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001671"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986360,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:38.306 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:38.306 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73312, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:12.337906+0000', 'last_change': '2026-02-12T04:13:12.337906+0000', 'last_active': '2026-02-12T04:13:12.337906+0000', 'last_peered': '2026-02-12T04:13:12.337906+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:12.337906+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:38.306 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:38.526 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:38.526 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1142,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:38.536 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:38.536 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:38.714 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:38.715 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:38.724 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2026-02-12T04:13:37.805970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001671"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986360,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:41.725 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:41.903 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:41.904 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:41.912 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":598,"stamp":"2026-02-12T04:13:41.806540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001657"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:41.913 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:42.091 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:42.091 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:42.100 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":598,"stamp":"2026-02-12T04:13:41.806540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001657"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:42.101 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:42.101 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73312, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:12.337906+0000', 'last_change': '2026-02-12T04:13:12.337906+0000', 'last_active': '2026-02-12T04:13:12.337906+0000', 'last_peered': '2026-02-12T04:13:12.337906+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:12.337906+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:42.102 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:42.323 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:42.323 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1145,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:42.332 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:42.332 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:42.508 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:42.508 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:42.516 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":598,"stamp":"2026-02-12T04:13:41.806540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001657"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73312,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:12.337906+0000","last_change":"2026-02-12T04:13:12.337906+0000","last_active":"2026-02-12T04:13:12.337906+0000","last_peered":"2026-02-12T04:13:12.337906+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:12.337906+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:45.518 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:45.703 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:45.704 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:45.713 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":599,"stamp":"2026-02-12T04:13:43.806786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001653"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:45.714 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:45.890 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:45.891 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:45.900 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":600,"stamp":"2026-02-12T04:13:45.807089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001670"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986362,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:45.901 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:45.901 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73316, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:42.338379+0000', 'last_change': '2026-02-12T04:13:42.338379+0000', 'last_active': '2026-02-12T04:13:42.338379+0000', 'last_peered': '2026-02-12T04:13:42.338379+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:42.338379+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:45.902 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:46.126 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:46.126 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1149,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:46.135 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:46.136 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:46.317 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:46.317 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:46.326 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":600,"stamp":"2026-02-12T04:13:45.807089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001670"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986362,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:49.328 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:49.507 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:49.508 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:49.517 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":601,"stamp":"2026-02-12T04:13:47.807347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001666"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986362,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:49.518 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:49.693 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:49.694 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:49.702 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":601,"stamp":"2026-02-12T04:13:47.807347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001666"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986362,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:49.703 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:49.703 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73316, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:42.338379+0000', 'last_change': '2026-02-12T04:13:42.338379+0000', 'last_active': '2026-02-12T04:13:42.338379+0000', 'last_peered': '2026-02-12T04:13:42.338379+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:42.338379+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:49.704 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:49.926 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:49.926 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1153,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:49.936 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:49.936 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:50.116 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:50.116 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:50.126 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":602,"stamp":"2026-02-12T04:13:49.807635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001664"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986362,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:53.128 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:53.311 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:53.311 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:53.320 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":603,"stamp":"2026-02-12T04:13:51.807897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001647"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986363,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:53.321 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:53.498 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:53.498 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:53.507 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":603,"stamp":"2026-02-12T04:13:51.807897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001647"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986363,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:53.508 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:53.508 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73316, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:42.338379+0000', 'last_change': '2026-02-12T04:13:42.338379+0000', 'last_active': '2026-02-12T04:13:42.338379+0000', 'last_peered': '2026-02-12T04:13:42.338379+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:42.338379+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:53.509 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:53.730 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:53.730 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1157,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:53.739 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:53.739 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:54.011 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:54.012 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:54.021 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":604,"stamp":"2026-02-12T04:13:53.808210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001669"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986363,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:57.023 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:57.203 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:57.204 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:57.213 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":605,"stamp":"2026-02-12T04:13:55.808460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001673"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986364,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:57.214 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:57.395 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:57.396 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:57.405 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":605,"stamp":"2026-02-12T04:13:55.808460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001673"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986364,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:13:57.406 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:13:57.406 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73316, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:42.338379+0000', 'last_change': '2026-02-12T04:13:42.338379+0000', 'last_active': '2026-02-12T04:13:42.338379+0000', 'last_peered': '2026-02-12T04:13:42.338379+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:42.338379+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:13:57.406 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:13:57.628 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:57.628 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1161,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:13:57.638 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:13:57.638 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:13:57.816 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:13:57.816 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:13:57.825 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":606,"stamp":"2026-02-12T04:13:57.808741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001651"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986364,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:00.827 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:01.009 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:01.010 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:01.019 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":607,"stamp":"2026-02-12T04:13:59.808982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001634"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986364,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:01.020 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:01.200 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:01.200 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:01.209 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":607,"stamp":"2026-02-12T04:13:59.808982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001634"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986364,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:01.210 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:01.210 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73316, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:42.338379+0000', 'last_change': '2026-02-12T04:13:42.338379+0000', 'last_active': '2026-02-12T04:13:42.338379+0000', 'last_peered': '2026-02-12T04:13:42.338379+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:42.338379+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:01.210 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:01.432 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:01.433 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1164,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:14:01.442 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:01.442 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:01.622 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:01.622 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:01.630 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":607,"stamp":"2026-02-12T04:13:59.808982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001634"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986364,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:04.632 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:04.812 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:04.812 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:04.821 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2026-02-12T04:14:03.809442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001545"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986365,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:04.822 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:04.998 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:04.999 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:05.008 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2026-02-12T04:14:03.809442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001545"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986365,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:05.009 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:05.009 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73316, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:42.338379+0000', 'last_change': '2026-02-12T04:13:42.338379+0000', 'last_active': '2026-02-12T04:13:42.338379+0000', 'last_peered': '2026-02-12T04:13:42.338379+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:42.338379+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:05.010 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:05.231 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:05.231 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1168,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:14:05.240 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:05.240 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:05.416 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:05.417 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:05.425 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2026-02-12T04:14:03.809442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001545"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986365,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:08.427 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:08.701 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:14:08.700+0000 7f773a27c640 0 --2- 10.20.193.103:0/1160344616 >> [v2:10.20.193.103:3302/0,v1:10.20.193.103:6791/0] conn(0x7f7734134120 0x7f773410cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:14:08.808 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:08.809 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:08.817 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":611,"stamp":"2026-02-12T04:14:07.809916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986366,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:08.818 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:08.997 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:08.998 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:09.007 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":611,"stamp":"2026-02-12T04:14:07.809916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986366,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:09.008 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:09.008 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73316, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:42.338379+0000', 'last_change': '2026-02-12T04:13:42.338379+0000', 'last_active': '2026-02-12T04:13:42.338379+0000', 'last_peered': '2026-02-12T04:13:42.338379+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:42.338379+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:09.009 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:09.236 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:09.236 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1172,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:14:09.245 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:09.245 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:09.421 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:09.421 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:09.430 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":611,"stamp":"2026-02-12T04:14:07.809916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986366,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:12.431 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:12.608 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:12.608 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:12.617 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":613,"stamp":"2026-02-12T04:14:11.810523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:12.618 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:12.810 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:12.810 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:12.819 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":613,"stamp":"2026-02-12T04:14:11.810523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:12.820 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:12.820 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73316, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:13:42.338379+0000', 'last_change': '2026-02-12T04:13:42.338379+0000', 'last_active': '2026-02-12T04:13:42.338379+0000', 'last_peered': '2026-02-12T04:13:42.338379+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:13:42.338379+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:12.821 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:13.040 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:13.040 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1176,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":18,"modified":"2026-02-12T04:12:27.796274+0000","services":{}},"progress_events":{}} 2026-02-12T04:14:13.050 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:13.051 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:13.235 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:13.235 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:13.244 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":613,"stamp":"2026-02-12T04:14:11.810523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73316,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:13:42.338379+0000","last_change":"2026-02-12T04:13:42.338379+0000","last_active":"2026-02-12T04:13:42.338379+0000","last_peered":"2026-02-12T04:13:42.338379+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:13:42.338379+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:16.246 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:16.424 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:16.425 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:16.433 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":615,"stamp":"2026-02-12T04:14:15.811148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001705"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986368,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:16.435 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:16.612 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:16.613 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:16.622 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":615,"stamp":"2026-02-12T04:14:15.811148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001705"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986368,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:16.623 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:16.623 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73320, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:12.338842+0000', 'last_change': '2026-02-12T04:14:12.338842+0000', 'last_active': '2026-02-12T04:14:12.338842+0000', 'last_peered': '2026-02-12T04:14:12.338842+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:12.338842+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:16.623 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:16.845 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:16.845 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1180,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":19,"modified":"2026-02-12T04:14:13.810887+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:14:16.854 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:16.855 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:17.033 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:17.034 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:17.043 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":615,"stamp":"2026-02-12T04:14:15.811148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001705"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986368,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:20.045 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:20.227 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:20.228 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:20.236 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":617,"stamp":"2026-02-12T04:14:19.811611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001694"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986368,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:20.237 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:20.417 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:20.418 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:20.426 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":617,"stamp":"2026-02-12T04:14:19.811611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001694"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986368,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:20.427 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:20.427 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73320, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:12.338842+0000', 'last_change': '2026-02-12T04:14:12.338842+0000', 'last_active': '2026-02-12T04:14:12.338842+0000', 'last_peered': '2026-02-12T04:14:12.338842+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:12.338842+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:20.427 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:20.643 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:20.643 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1184,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":19,"modified":"2026-02-12T04:14:13.810887+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:14:20.652 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:20.653 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:20.831 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:20.832 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:20.841 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":617,"stamp":"2026-02-12T04:14:19.811611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001694"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986368,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:23.843 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:24.024 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:24.025 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:24.034 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":619,"stamp":"2026-02-12T04:14:23.812156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986369,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:24.035 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:24.212 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:24.213 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:24.221 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":619,"stamp":"2026-02-12T04:14:23.812156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986369,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:24.222 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:24.223 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73320, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:12.338842+0000', 'last_change': '2026-02-12T04:14:12.338842+0000', 'last_active': '2026-02-12T04:14:12.338842+0000', 'last_peered': '2026-02-12T04:14:12.338842+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:12.338842+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:24.223 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:24.444 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:24.444 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1187,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":19,"modified":"2026-02-12T04:14:13.810887+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:14:24.454 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:24.454 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:24.631 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:24.632 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:24.640 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":619,"stamp":"2026-02-12T04:14:23.812156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986369,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:27.642 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:27.820 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:27.821 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:27.829 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":621,"stamp":"2026-02-12T04:14:27.812724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001575"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986370,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:27.831 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:28.009 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:28.009 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:28.019 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":621,"stamp":"2026-02-12T04:14:27.812724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001575"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986370,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:28.020 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:28.020 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73320, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:12.338842+0000', 'last_change': '2026-02-12T04:14:12.338842+0000', 'last_active': '2026-02-12T04:14:12.338842+0000', 'last_peered': '2026-02-12T04:14:12.338842+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:12.338842+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:28.020 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:28.243 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:28.243 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1191,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":19,"modified":"2026-02-12T04:14:13.810887+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:14:28.252 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:28.252 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:28.432 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:28.432 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:28.441 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":621,"stamp":"2026-02-12T04:14:27.812724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001575"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986370,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:31.443 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:31.629 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:31.637 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:31.638 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":622,"stamp":"2026-02-12T04:14:29.812974+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001600"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986370,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:31.639 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:31.817 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:31.818 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:31.827 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":623,"stamp":"2026-02-12T04:14:31.813267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001656"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:31.828 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:31.828 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73320, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:12.338842+0000', 'last_change': '2026-02-12T04:14:12.338842+0000', 'last_active': '2026-02-12T04:14:12.338842+0000', 'last_peered': '2026-02-12T04:14:12.338842+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:12.338842+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:31.829 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:32.051 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:32.051 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1195,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":19,"modified":"2026-02-12T04:14:13.810887+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:14:32.061 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:32.061 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:32.240 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:32.241 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:32.250 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":623,"stamp":"2026-02-12T04:14:31.813267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001656"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:35.252 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:35.428 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:35.429 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:35.437 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":624,"stamp":"2026-02-12T04:14:33.813547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001690"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:35.438 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:35.616 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:35.616 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:35.624 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":624,"stamp":"2026-02-12T04:14:33.813547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001690"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:35.625 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:35.626 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73320, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:12.338842+0000', 'last_change': '2026-02-12T04:14:12.338842+0000', 'last_active': '2026-02-12T04:14:12.338842+0000', 'last_peered': '2026-02-12T04:14:12.338842+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:12.338842+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:35.626 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:35.849 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:35.850 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1199,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":20,"modified":"2026-02-12T04:14:33.813607+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:14:35.860 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:35.860 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:36.038 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:36.038 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:36.047 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":625,"stamp":"2026-02-12T04:14:35.813889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001733"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:39.049 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:39.225 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:39.226 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:39.234 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":626,"stamp":"2026-02-12T04:14:37.814175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001755"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:39.235 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:39.412 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:39.412 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:39.421 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":626,"stamp":"2026-02-12T04:14:37.814175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001755"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:39.422 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:39.422 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73320, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:12.338842+0000', 'last_change': '2026-02-12T04:14:12.338842+0000', 'last_active': '2026-02-12T04:14:12.338842+0000', 'last_peered': '2026-02-12T04:14:12.338842+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:12.338842+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:39.423 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:39.644 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:39.645 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1203,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":20,"modified":"2026-02-12T04:14:33.813607+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:14:39.655 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:39.655 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:39.831 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:39.832 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:39.840 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":627,"stamp":"2026-02-12T04:14:39.814506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001782"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:10:40.267270+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:10:40.267270+0000","last_peered":"2026-02-12T04:10:40.267270+0000","last_clean":"2026-02-12T04:10:40.267270+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:10:40.267270+0000","last_undegraded":"2026-02-12T04:10:40.267270+0000","last_fullsized":"2026-02-12T04:10:40.267270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:42.842 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:43.021 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:43.021 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:43.031 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":628,"stamp":"2026-02-12T04:14:41.814761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001787"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:43.032 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:43.208 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:43.209 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:43.217 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":628,"stamp":"2026-02-12T04:14:41.814761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001787"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:43.218 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:43.218 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73320, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:12.338842+0000', 'last_change': '2026-02-12T04:14:12.338842+0000', 'last_active': '2026-02-12T04:14:12.338842+0000', 'last_peered': '2026-02-12T04:14:12.338842+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:12.338842+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:43.219 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:43.437 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:43.437 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1206,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":20,"modified":"2026-02-12T04:14:33.813607+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:14:43.447 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:43.447 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:43.628 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:43.628 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:43.638 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":628,"stamp":"2026-02-12T04:14:41.814761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001787"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73320,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:12.338842+0000","last_change":"2026-02-12T04:14:12.338842+0000","last_active":"2026-02-12T04:14:12.338842+0000","last_peered":"2026-02-12T04:14:12.338842+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:12.338842+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:46.639 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:46.823 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:46.823 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:46.832 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":630,"stamp":"2026-02-12T04:14:45.815276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001728"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73324,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339229+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339229+0000","last_peered":"2026-02-12T04:14:42.339229+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339229+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:46.833 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:47.009 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:47.009 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:47.018 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":630,"stamp":"2026-02-12T04:14:45.815276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001728"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73324,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339229+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339229+0000","last_peered":"2026-02-12T04:14:42.339229+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339229+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:47.019 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:47.019 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73324, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:42.339229+0000', 'last_change': '2026-02-12T04:14:42.339229+0000', 'last_active': '2026-02-12T04:14:42.339229+0000', 'last_peered': '2026-02-12T04:14:42.339229+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:42.339229+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:47.019 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:47.250 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:47.250 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1210,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":20,"modified":"2026-02-12T04:14:33.813607+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:14:47.259 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:47.260 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:47.438 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:47.439 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:47.448 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":630,"stamp":"2026-02-12T04:14:45.815276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001728"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73324,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339229+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339229+0000","last_peered":"2026-02-12T04:14:42.339229+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339229+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:50.450 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:50.631 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:50.631 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:50.641 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":632,"stamp":"2026-02-12T04:14:49.815868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001693"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73324,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339229+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339229+0000","last_peered":"2026-02-12T04:14:42.339229+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339229+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:50.642 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:50.819 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:50.819 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:50.828 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":632,"stamp":"2026-02-12T04:14:49.815868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001693"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73324,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339229+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339229+0000","last_peered":"2026-02-12T04:14:42.339229+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339229+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:50.829 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:50.829 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73324, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:42.339229+0000', 'last_change': '2026-02-12T04:14:42.339229+0000', 'last_active': '2026-02-12T04:14:42.339229+0000', 'last_peered': '2026-02-12T04:14:42.339229+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:42.339229+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:50.830 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:51.050 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:51.050 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1214,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":20,"modified":"2026-02-12T04:14:33.813607+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:14:51.060 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:51.061 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:51.235 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:51.236 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:51.245 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":632,"stamp":"2026-02-12T04:14:49.815868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001693"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73324,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339229+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339229+0000","last_peered":"2026-02-12T04:14:42.339229+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339229+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:54.247 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:54.426 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:54.426 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:54.435 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":634,"stamp":"2026-02-12T04:14:53.816409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001648"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73324,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339229+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339229+0000","last_peered":"2026-02-12T04:14:42.339229+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339229+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:54.436 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:54.612 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:54.612 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:54.621 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":634,"stamp":"2026-02-12T04:14:53.816409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001648"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73324,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339229+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339229+0000","last_peered":"2026-02-12T04:14:42.339229+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339229+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:54.622 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:54.622 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73324, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:42.339229+0000', 'last_change': '2026-02-12T04:14:42.339229+0000', 'last_active': '2026-02-12T04:14:42.339229+0000', 'last_peered': '2026-02-12T04:14:42.339229+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:42.339229+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:54.622 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:54.847 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:54.847 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1218,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":20,"modified":"2026-02-12T04:14:33.813607+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:14:54.856 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:54.856 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:55.032 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:55.032 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:55.040 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":634,"stamp":"2026-02-12T04:14:53.816409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001648"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73324,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339229+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339229+0000","last_peered":"2026-02-12T04:14:42.339229+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339229+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:58.041 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:58.217 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:58.218 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:58.226 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2026-02-12T04:14:57.816996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001720"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73325,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339266+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339266+0000","last_peered":"2026-02-12T04:14:42.339266+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339266+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986376,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:58.226 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:58.404 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:58.404 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:58.413 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2026-02-12T04:14:57.816996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001720"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73325,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339266+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339266+0000","last_peered":"2026-02-12T04:14:42.339266+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339266+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986376,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:14:58.414 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:14:58.414 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73325, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:42.339266+0000', 'last_change': '2026-02-12T04:14:42.339229+0000', 'last_active': '2026-02-12T04:14:42.339266+0000', 'last_peered': '2026-02-12T04:14:42.339266+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:42.339266+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:14:58.414 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:14:58.636 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:58.636 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1222,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":20,"modified":"2026-02-12T04:14:33.813607+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:14:58.645 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:14:58.645 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:14:58.823 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:14:58.824 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:14:58.833 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2026-02-12T04:14:57.816996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001720"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73325,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339266+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339266+0000","last_peered":"2026-02-12T04:14:42.339266+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339266+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986376,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:01.834 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:02.010 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:02.010 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:02.019 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":638,"stamp":"2026-02-12T04:15:01.817609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001741"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73325,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339266+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339266+0000","last_peered":"2026-02-12T04:14:42.339266+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339266+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:02.019 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:02.195 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:02.196 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:02.205 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":638,"stamp":"2026-02-12T04:15:01.817609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001741"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73325,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339266+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339266+0000","last_peered":"2026-02-12T04:14:42.339266+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339266+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:02.206 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:02.206 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73325, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:42.339266+0000', 'last_change': '2026-02-12T04:14:42.339229+0000', 'last_active': '2026-02-12T04:14:42.339266+0000', 'last_peered': '2026-02-12T04:14:42.339266+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:42.339266+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:02.206 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:02.426 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:02.426 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1225,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":20,"modified":"2026-02-12T04:14:33.813607+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:15:02.435 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:02.435 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:02.610 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:02.610 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:02.618 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":638,"stamp":"2026-02-12T04:15:01.817609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001741"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73325,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339266+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339266+0000","last_peered":"2026-02-12T04:14:42.339266+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339266+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:05.620 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:05.800 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:05.800 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:05.809 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":639,"stamp":"2026-02-12T04:15:03.817916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001812"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73325,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339266+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339266+0000","last_peered":"2026-02-12T04:14:42.339266+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339266+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:05.810 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:05.992 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:05.993 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:06.002 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":640,"stamp":"2026-02-12T04:15:05.818221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001812"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73325,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339266+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339266+0000","last_peered":"2026-02-12T04:14:42.339266+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339266+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986378,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:06.003 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:06.003 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73325, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:42.339266+0000', 'last_change': '2026-02-12T04:14:42.339229+0000', 'last_active': '2026-02-12T04:14:42.339266+0000', 'last_peered': '2026-02-12T04:14:42.339266+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:42.339266+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:06.004 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:06.271 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:15:06.270+0000 7fc7e1986640 0 --2- 10.20.193.103:0/929901650 >> [v2:10.20.193.103:3301/0,v1:10.20.193.103:6790/0] conn(0x7fc7dc11a190 0x7fc7dc11a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:15:06.426 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:06.426 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1229,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":20,"modified":"2026-02-12T04:14:33.813607+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:15:06.436 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:06.437 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:06.616 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:06.617 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:06.626 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":640,"stamp":"2026-02-12T04:15:05.818221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001812"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73325,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339266+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339266+0000","last_peered":"2026-02-12T04:14:42.339266+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339266+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986378,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:09.628 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:09.812 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:09.813 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:09.821 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":641,"stamp":"2026-02-12T04:15:07.818477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001783"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73325,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339266+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339266+0000","last_peered":"2026-02-12T04:14:42.339266+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339266+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986378,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:09.822 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:10.002 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:10.002 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:10.011 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":642,"stamp":"2026-02-12T04:15:09.818773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001776"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73325,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339266+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339266+0000","last_peered":"2026-02-12T04:14:42.339266+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339266+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986378,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:10.012 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:10.013 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73325, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:42.339266+0000', 'last_change': '2026-02-12T04:14:42.339229+0000', 'last_active': '2026-02-12T04:14:42.339266+0000', 'last_peered': '2026-02-12T04:14:42.339266+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:42.339266+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:10.013 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:10.233 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:10.234 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1233,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:15:10.243 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:10.243 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:10.423 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:10.424 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:10.434 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":642,"stamp":"2026-02-12T04:15:09.818773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001776"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73325,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339266+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339266+0000","last_peered":"2026-02-12T04:14:42.339266+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339266+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986378,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:13.436 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:13.613 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:13.614 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:13.621 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":643,"stamp":"2026-02-12T04:15:11.819012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001709"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73325,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339266+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339266+0000","last_peered":"2026-02-12T04:14:42.339266+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339266+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986379,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:13.622 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:13.796 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:13.796 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:13.804 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":643,"stamp":"2026-02-12T04:15:11.819012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001709"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73325,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:14:42.339266+0000","last_change":"2026-02-12T04:14:42.339229+0000","last_active":"2026-02-12T04:14:42.339266+0000","last_peered":"2026-02-12T04:14:42.339266+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:14:42.339266+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986379,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:13.805 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:13.806 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73325, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:14:42.339266+0000', 'last_change': '2026-02-12T04:14:42.339229+0000', 'last_active': '2026-02-12T04:14:42.339266+0000', 'last_peered': '2026-02-12T04:14:42.339266+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:14:42.339266+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:13.806 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:14.026 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:14.026 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1237,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:15:14.036 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:14.036 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:14.214 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:14.215 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:14.224 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":644,"stamp":"2026-02-12T04:15:13.819303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001693"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986379,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:17.226 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:17.407 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:17.407 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:17.417 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":645,"stamp":"2026-02-12T04:15:15.819539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986380,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:17.417 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:17.593 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:17.593 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:17.601 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":645,"stamp":"2026-02-12T04:15:15.819539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986380,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:17.602 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:17.602 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73329, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:12.339654+0000', 'last_change': '2026-02-12T04:15:12.339654+0000', 'last_active': '2026-02-12T04:15:12.339654+0000', 'last_peered': '2026-02-12T04:15:12.339654+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:12.339654+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:17.602 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:17.819 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:17.819 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1241,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:15:17.828 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:17.828 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:18.004 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:18.005 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:18.013 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":646,"stamp":"2026-02-12T04:15:17.819830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001608"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986380,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:21.015 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:21.196 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:21.197 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:21.206 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":647,"stamp":"2026-02-12T04:15:19.820046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001569"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986380,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:21.207 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:21.385 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:21.385 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:21.395 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":647,"stamp":"2026-02-12T04:15:19.820046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001569"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986380,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:21.396 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:21.396 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73329, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:12.339654+0000', 'last_change': '2026-02-12T04:15:12.339654+0000', 'last_active': '2026-02-12T04:15:12.339654+0000', 'last_peered': '2026-02-12T04:15:12.339654+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:12.339654+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:21.396 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:21.618 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:21.619 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1245,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:15:21.628 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:21.628 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:21.803 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:21.803 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:21.811 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":647,"stamp":"2026-02-12T04:15:19.820046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001569"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986380,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:24.813 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:24.993 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:24.993 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:25.002 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":649,"stamp":"2026-02-12T04:15:23.820534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001522"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986381,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:25.003 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:25.186 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:25.186 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:25.195 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":649,"stamp":"2026-02-12T04:15:23.820534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001522"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986381,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:25.196 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:25.196 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73329, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:12.339654+0000', 'last_change': '2026-02-12T04:15:12.339654+0000', 'last_active': '2026-02-12T04:15:12.339654+0000', 'last_peered': '2026-02-12T04:15:12.339654+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:12.339654+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:25.196 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:25.418 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:25.419 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1248,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:15:25.428 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:25.428 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:25.603 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:25.604 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:25.612 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":649,"stamp":"2026-02-12T04:15:23.820534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001522"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986381,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:28.614 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:28.793 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:28.793 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:28.802 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":651,"stamp":"2026-02-12T04:15:27.821026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001487"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986382,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:28.803 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:29.076 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:29.076 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:29.085 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":651,"stamp":"2026-02-12T04:15:27.821026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001487"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986382,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:29.086 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:29.086 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73329, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:12.339654+0000', 'last_change': '2026-02-12T04:15:12.339654+0000', 'last_active': '2026-02-12T04:15:12.339654+0000', 'last_peered': '2026-02-12T04:15:12.339654+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:12.339654+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:29.086 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:29.305 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:29.305 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1252,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:15:29.314 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:29.314 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:29.491 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:29.491 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:29.500 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":651,"stamp":"2026-02-12T04:15:27.821026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001487"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986382,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:32.502 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:32.681 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:32.682 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:32.690 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":653,"stamp":"2026-02-12T04:15:31.821490+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001444"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986383,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:32.692 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:32.870 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:32.870 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:32.879 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":653,"stamp":"2026-02-12T04:15:31.821490+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001444"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986383,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:32.880 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:32.880 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73329, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:12.339654+0000', 'last_change': '2026-02-12T04:15:12.339654+0000', 'last_active': '2026-02-12T04:15:12.339654+0000', 'last_peered': '2026-02-12T04:15:12.339654+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:12.339654+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:32.881 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:33.108 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:33.108 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1256,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:15:33.118 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:33.118 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:33.297 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:33.298 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:33.306 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":653,"stamp":"2026-02-12T04:15:31.821490+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001444"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986383,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:36.308 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:36.488 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:36.488 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:36.498 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":655,"stamp":"2026-02-12T04:15:35.822091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001556"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:36.499 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:36.769 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:15:36.768+0000 7f1722176640 0 --2- 10.20.193.103:0/378637221 >> [v2:10.20.193.103:3302/0,v1:10.20.193.103:6791/0] conn(0x7f171c12d870 0x7f171c12dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:15:36.883 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:36.883 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:36.892 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":655,"stamp":"2026-02-12T04:15:35.822091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001556"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:36.893 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:36.894 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73329, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:12.339654+0000', 'last_change': '2026-02-12T04:15:12.339654+0000', 'last_active': '2026-02-12T04:15:12.339654+0000', 'last_peered': '2026-02-12T04:15:12.339654+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:12.339654+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:36.894 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:37.114 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:37.114 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1260,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:15:37.124 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:37.124 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:37.303 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:37.304 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:37.313 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":655,"stamp":"2026-02-12T04:15:35.822091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001556"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:40.315 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:40.498 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:40.498 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:40.507 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":657,"stamp":"2026-02-12T04:15:39.822519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001492"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:40.508 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:40.686 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:40.687 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:40.695 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":657,"stamp":"2026-02-12T04:15:39.822519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001492"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:40.696 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:40.696 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73329, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:12.339654+0000', 'last_change': '2026-02-12T04:15:12.339654+0000', 'last_active': '2026-02-12T04:15:12.339654+0000', 'last_peered': '2026-02-12T04:15:12.339654+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:12.339654+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:40.697 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:40.922 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:40.923 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1264,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:15:40.933 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:40.933 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:41.114 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:41.115 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:41.123 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":657,"stamp":"2026-02-12T04:15:39.822519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001492"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73329,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:12.339654+0000","last_change":"2026-02-12T04:15:12.339654+0000","last_active":"2026-02-12T04:15:12.339654+0000","last_peered":"2026-02-12T04:15:12.339654+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:12.339654+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:44.124 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:44.314 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:44.314 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:44.324 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2026-02-12T04:15:43.823040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001549"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986385,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:44.325 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:44.507 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:44.507 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:44.518 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2026-02-12T04:15:43.823040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001549"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986385,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:44.519 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:44.519 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73333, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:42.340250+0000', 'last_change': '2026-02-12T04:15:42.340250+0000', 'last_active': '2026-02-12T04:15:42.340250+0000', 'last_peered': '2026-02-12T04:15:42.340250+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:42.340250+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:44.519 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:44.745 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:44.745 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1268,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:15:44.755 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:44.755 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:44.934 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:44.934 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:44.943 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2026-02-12T04:15:43.823040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001549"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986385,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:47.945 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:48.123 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:48.123 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:48.132 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":661,"stamp":"2026-02-12T04:15:47.823604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986386,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:48.133 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:48.309 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:48.310 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:48.318 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":661,"stamp":"2026-02-12T04:15:47.823604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986386,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:48.319 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:48.319 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73333, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:42.340250+0000', 'last_change': '2026-02-12T04:15:42.340250+0000', 'last_active': '2026-02-12T04:15:42.340250+0000', 'last_peered': '2026-02-12T04:15:42.340250+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:42.340250+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:48.320 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:48.537 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:48.538 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1272,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:15:48.548 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:48.548 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:48.723 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:48.724 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:48.732 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":661,"stamp":"2026-02-12T04:15:47.823604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986386,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:51.734 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:52.007 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:52.007 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:52.017 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":663,"stamp":"2026-02-12T04:15:51.824133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001613"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986387,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:52.018 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:52.196 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:52.196 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:52.205 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":663,"stamp":"2026-02-12T04:15:51.824133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001613"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986387,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:52.206 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:52.207 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73333, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:42.340250+0000', 'last_change': '2026-02-12T04:15:42.340250+0000', 'last_active': '2026-02-12T04:15:42.340250+0000', 'last_peered': '2026-02-12T04:15:42.340250+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:42.340250+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:52.207 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:52.425 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:52.425 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1275,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:15:52.434 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:52.434 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:52.613 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:52.614 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:52.622 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":663,"stamp":"2026-02-12T04:15:51.824133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001613"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986387,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:55.624 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:55.807 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:55.807 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:55.816 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":664,"stamp":"2026-02-12T04:15:53.824369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986387,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:55.817 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:55.991 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:55.992 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:56.001 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":665,"stamp":"2026-02-12T04:15:55.824578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001538"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986388,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:56.002 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:15:56.003 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73333, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:42.340250+0000', 'last_change': '2026-02-12T04:15:42.340250+0000', 'last_active': '2026-02-12T04:15:42.340250+0000', 'last_peered': '2026-02-12T04:15:42.340250+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:42.340250+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:15:56.003 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:15:56.222 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:56.222 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1279,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:15:56.232 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:15:56.232 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:56.406 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:56.406 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:56.415 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":665,"stamp":"2026-02-12T04:15:55.824578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001538"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986388,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:59.416 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:15:59.688 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:15:59.686+0000 7f4c3f7fe640 0 --2- 10.20.193.103:0/2341945510 >> [v2:10.20.193.103:3302/0,v1:10.20.193.103:6791/0] conn(0x7f4c40154e40 0x7f4c40175220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:15:59.801 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:15:59.801 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:15:59.812 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":666,"stamp":"2026-02-12T04:15:57.824795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001512"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986388,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:15:59.813 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:00.086 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:00.086 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:00.096 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":667,"stamp":"2026-02-12T04:15:59.825034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001430"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986388,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:00.097 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:00.097 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73333, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:42.340250+0000', 'last_change': '2026-02-12T04:15:42.340250+0000', 'last_active': '2026-02-12T04:15:42.340250+0000', 'last_peered': '2026-02-12T04:15:42.340250+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:42.340250+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:00.097 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:00.316 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:00.316 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1283,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:00.326 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:00.326 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:00.506 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:00.506 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:00.516 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":667,"stamp":"2026-02-12T04:15:59.825034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001430"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986388,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:03.518 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:03.696 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:03.697 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:03.706 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2026-02-12T04:16:01.825282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001379"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986389,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:03.707 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:03.887 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:03.888 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:03.896 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":669,"stamp":"2026-02-12T04:16:03.825573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001440"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986389,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:03.897 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:03.897 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73333, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:42.340250+0000', 'last_change': '2026-02-12T04:15:42.340250+0000', 'last_active': '2026-02-12T04:15:42.340250+0000', 'last_peered': '2026-02-12T04:15:42.340250+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:42.340250+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:03.898 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:04.117 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:04.117 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1287,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:04.126 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:04.127 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:04.301 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:04.301 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:04.310 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":669,"stamp":"2026-02-12T04:16:03.825573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001440"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986389,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:07.312 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:07.492 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:07.492 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:07.503 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":670,"stamp":"2026-02-12T04:16:05.825817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001448"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986390,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:07.503 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:07.679 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:07.680 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:07.688 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":670,"stamp":"2026-02-12T04:16:05.825817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001448"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986390,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:07.689 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:07.689 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73333, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:42.340250+0000', 'last_change': '2026-02-12T04:15:42.340250+0000', 'last_active': '2026-02-12T04:15:42.340250+0000', 'last_peered': '2026-02-12T04:15:42.340250+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:42.340250+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:07.689 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:07.916 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:07.916 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1291,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:07.925 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:07.926 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:08.104 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:08.104 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:08.113 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":671,"stamp":"2026-02-12T04:16:07.826096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986390,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:11.115 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:11.291 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:11.292 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:11.300 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":672,"stamp":"2026-02-12T04:16:09.826327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001532"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986390,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:11.301 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:11.476 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:11.476 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:11.484 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":672,"stamp":"2026-02-12T04:16:09.826327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001532"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986390,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:11.485 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:11.486 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73333, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:15:42.340250+0000', 'last_change': '2026-02-12T04:15:42.340250+0000', 'last_active': '2026-02-12T04:15:42.340250+0000', 'last_peered': '2026-02-12T04:15:42.340250+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:15:42.340250+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:11.486 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:11.709 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:11.709 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1295,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:11.719 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:11.719 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:11.897 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:11.897 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:11.906 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":673,"stamp":"2026-02-12T04:16:11.826549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001515"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73333,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:15:42.340250+0000","last_change":"2026-02-12T04:15:42.340250+0000","last_active":"2026-02-12T04:15:42.340250+0000","last_peered":"2026-02-12T04:15:42.340250+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:15:42.340250+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986391,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:14.908 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:15.086 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:15.086 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:15.095 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":674,"stamp":"2026-02-12T04:16:13.826806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001524"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986391,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:15.096 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:15.274 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:15.274 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:15.283 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":674,"stamp":"2026-02-12T04:16:13.826806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001524"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986391,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:15.284 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:15.284 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73337, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:12.340752+0000', 'last_change': '2026-02-12T04:16:12.340752+0000', 'last_active': '2026-02-12T04:16:12.340752+0000', 'last_peered': '2026-02-12T04:16:12.340752+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:12.340752+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:15.284 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:15.506 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:15.507 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1299,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:15.517 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:15.517 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:15.788 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:16:15.787+0000 7fcb2e7fc640 0 --2- 10.20.193.103:0/2013646797 >> [v2:10.20.193.103:3300/0,v1:10.20.193.103:6789/0] conn(0x7fcb30130690 0x7fcb30120ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:16:16.791 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:16.791 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:16.804 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":675,"stamp":"2026-02-12T04:16:15.827053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001479"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986392,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:19.806 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:19.983 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:19.984 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:19.992 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":677,"stamp":"2026-02-12T04:16:19.827500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001404"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986392,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:19.993 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:20.168 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:20.169 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:20.177 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":677,"stamp":"2026-02-12T04:16:19.827500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001404"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986392,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:20.178 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:20.178 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73337, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:12.340752+0000', 'last_change': '2026-02-12T04:16:12.340752+0000', 'last_active': '2026-02-12T04:16:12.340752+0000', 'last_peered': '2026-02-12T04:16:12.340752+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:12.340752+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:20.178 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:20.401 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:20.401 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1303,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:20.411 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:20.411 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:20.588 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:20.589 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:20.598 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":677,"stamp":"2026-02-12T04:16:19.827500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001404"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986392,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:23.600 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:23.777 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:23.777 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:23.785 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":678,"stamp":"2026-02-12T04:16:21.827714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001387"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986393,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:23.786 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:23.963 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:23.964 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:23.972 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":679,"stamp":"2026-02-12T04:16:23.828012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001463"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986393,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:23.973 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:23.973 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73337, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:12.340752+0000', 'last_change': '2026-02-12T04:16:12.340752+0000', 'last_active': '2026-02-12T04:16:12.340752+0000', 'last_peered': '2026-02-12T04:16:12.340752+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:12.340752+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:23.974 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:24.193 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:24.193 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1307,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:24.202 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:24.202 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:24.382 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:24.383 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:24.392 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":679,"stamp":"2026-02-12T04:16:23.828012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001463"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986393,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:27.394 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:27.571 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:27.572 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:27.581 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":680,"stamp":"2026-02-12T04:16:25.828260+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001453"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986394,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:27.582 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:27.757 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:27.758 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:27.766 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":680,"stamp":"2026-02-12T04:16:25.828260+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001453"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986394,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:27.767 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:27.767 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73337, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:12.340752+0000', 'last_change': '2026-02-12T04:16:12.340752+0000', 'last_active': '2026-02-12T04:16:12.340752+0000', 'last_peered': '2026-02-12T04:16:12.340752+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:12.340752+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:27.768 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:27.988 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:27.988 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1311,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:27.998 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:27.998 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:28.173 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:28.173 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:28.182 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":681,"stamp":"2026-02-12T04:16:27.828546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986394,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:31.184 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:31.369 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:31.369 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:31.378 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":682,"stamp":"2026-02-12T04:16:29.828825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986394,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:31.379 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:31.646 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:16:31.645+0000 7f6f177fe640 0 --2- 10.20.193.103:0/2910883268 >> [v2:10.20.193.103:3301/0,v1:10.20.193.103:6790/0] conn(0x7f6f18175760 0x7f6f18195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:16:31.759 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:31.759 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:31.768 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":682,"stamp":"2026-02-12T04:16:29.828825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986394,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:31.769 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:31.769 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73337, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:12.340752+0000', 'last_change': '2026-02-12T04:16:12.340752+0000', 'last_active': '2026-02-12T04:16:12.340752+0000', 'last_peered': '2026-02-12T04:16:12.340752+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:12.340752+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:31.770 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:31.988 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:31.989 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1315,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:31.998 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:31.998 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:32.172 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:32.173 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:32.181 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":683,"stamp":"2026-02-12T04:16:31.829112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001612"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986395,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:35.182 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:35.365 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:35.365 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:35.374 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":684,"stamp":"2026-02-12T04:16:33.829379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001664"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986395,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:35.375 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:35.553 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:35.553 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:35.563 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":684,"stamp":"2026-02-12T04:16:33.829379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001664"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986395,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:35.564 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:35.564 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73337, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:12.340752+0000', 'last_change': '2026-02-12T04:16:12.340752+0000', 'last_active': '2026-02-12T04:16:12.340752+0000', 'last_peered': '2026-02-12T04:16:12.340752+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:12.340752+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:35.564 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:35.787 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:35.787 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1319,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:35.797 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:35.797 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:35.975 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:35.975 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:35.984 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":685,"stamp":"2026-02-12T04:16:35.829657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001644"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986396,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:38.986 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:39.165 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:39.166 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:39.174 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":686,"stamp":"2026-02-12T04:16:37.829914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001654"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986396,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:39.175 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:39.356 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:39.356 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:39.366 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":686,"stamp":"2026-02-12T04:16:37.829914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001654"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986396,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:39.367 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:39.367 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73337, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:12.340752+0000', 'last_change': '2026-02-12T04:16:12.340752+0000', 'last_active': '2026-02-12T04:16:12.340752+0000', 'last_peered': '2026-02-12T04:16:12.340752+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:12.340752+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:39.367 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:39.600 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:39.600 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1323,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:39.609 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:39.610 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:39.785 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:39.785 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:39.793 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":686,"stamp":"2026-02-12T04:16:37.829914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001654"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986396,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:42.795 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:42.974 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:42.974 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:42.983 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":688,"stamp":"2026-02-12T04:16:41.830457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986397,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:42.984 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:43.164 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:43.165 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:43.174 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":688,"stamp":"2026-02-12T04:16:41.830457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986397,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:43.175 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:43.175 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73337, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:12.340752+0000', 'last_change': '2026-02-12T04:16:12.340752+0000', 'last_active': '2026-02-12T04:16:12.340752+0000', 'last_peered': '2026-02-12T04:16:12.340752+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:12.340752+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:43.175 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:43.395 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:43.396 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1326,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:43.405 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:43.406 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:43.581 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:43.582 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:43.591 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":688,"stamp":"2026-02-12T04:16:41.830457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73337,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:12.340752+0000","last_change":"2026-02-12T04:16:12.340752+0000","last_active":"2026-02-12T04:16:12.340752+0000","last_peered":"2026-02-12T04:16:12.340752+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:12.340752+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:12:40.277697+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:12:40.277697+0000","last_peered":"2026-02-12T04:12:40.277697+0000","last_clean":"2026-02-12T04:12:40.277697+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:12:40.277697+0000","last_undegraded":"2026-02-12T04:12:40.277697+0000","last_fullsized":"2026-02-12T04:12:40.277697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986397,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:46.593 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:46.778 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:46.779 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:46.787 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":690,"stamp":"2026-02-12T04:16:45.830985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001605"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73341,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341086+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341086+0000","last_peered":"2026-02-12T04:16:42.341086+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341086+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986398,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:46.788 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:46.967 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:46.968 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:46.977 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":690,"stamp":"2026-02-12T04:16:45.830985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001605"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73341,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341086+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341086+0000","last_peered":"2026-02-12T04:16:42.341086+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341086+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986398,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:46.978 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:46.978 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73341, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:42.341086+0000', 'last_change': '2026-02-12T04:16:42.341086+0000', 'last_active': '2026-02-12T04:16:42.341086+0000', 'last_peered': '2026-02-12T04:16:42.341086+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:42.341086+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:46.979 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:47.201 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:47.202 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1330,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:47.211 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:47.211 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:47.395 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:47.395 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:47.404 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":690,"stamp":"2026-02-12T04:16:45.830985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001605"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73341,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341086+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341086+0000","last_peered":"2026-02-12T04:16:42.341086+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341086+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986398,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:50.406 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:50.587 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:50.587 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:50.596 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":692,"stamp":"2026-02-12T04:16:49.831371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73341,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341086+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341086+0000","last_peered":"2026-02-12T04:16:42.341086+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341086+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986398,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:50.597 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:50.774 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:50.775 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:50.784 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":692,"stamp":"2026-02-12T04:16:49.831371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73341,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341086+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341086+0000","last_peered":"2026-02-12T04:16:42.341086+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341086+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986398,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:50.785 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:50.785 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73341, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:42.341086+0000', 'last_change': '2026-02-12T04:16:42.341086+0000', 'last_active': '2026-02-12T04:16:42.341086+0000', 'last_peered': '2026-02-12T04:16:42.341086+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:42.341086+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:50.785 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:51.006 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:51.006 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1334,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:51.016 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:51.017 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:51.195 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:51.195 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:51.204 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":692,"stamp":"2026-02-12T04:16:49.831371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73341,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341086+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341086+0000","last_peered":"2026-02-12T04:16:42.341086+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341086+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986398,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:54.206 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:54.386 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:54.387 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:54.395 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":694,"stamp":"2026-02-12T04:16:53.831845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001387"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73341,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341086+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341086+0000","last_peered":"2026-02-12T04:16:42.341086+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341086+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986399,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:54.396 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:54.575 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:54.575 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:54.584 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":694,"stamp":"2026-02-12T04:16:53.831845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001387"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73341,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341086+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341086+0000","last_peered":"2026-02-12T04:16:42.341086+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341086+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986399,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:54.585 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:54.585 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73341, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:42.341086+0000', 'last_change': '2026-02-12T04:16:42.341086+0000', 'last_active': '2026-02-12T04:16:42.341086+0000', 'last_peered': '2026-02-12T04:16:42.341086+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:42.341086+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:54.585 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:54.813 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:54.814 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1338,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:54.823 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:54.823 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:55.000 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:55.001 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:55.009 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":694,"stamp":"2026-02-12T04:16:53.831845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001387"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73341,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341086+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341086+0000","last_peered":"2026-02-12T04:16:42.341086+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341086+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986399,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:58.011 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:58.190 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:58.191 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:58.199 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":696,"stamp":"2026-02-12T04:16:57.832319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001333"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73342,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341111+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341111+0000","last_peered":"2026-02-12T04:16:42.341111+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341111+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986400,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:58.200 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:58.379 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:58.380 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:58.388 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":696,"stamp":"2026-02-12T04:16:57.832319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001333"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73342,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341111+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341111+0000","last_peered":"2026-02-12T04:16:42.341111+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341111+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986400,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:16:58.389 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:16:58.390 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73342, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:42.341111+0000', 'last_change': '2026-02-12T04:16:42.341086+0000', 'last_active': '2026-02-12T04:16:42.341111+0000', 'last_peered': '2026-02-12T04:16:42.341111+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:42.341111+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:16:58.390 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:16:58.612 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:58.613 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1342,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:16:58.621 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:16:58.622 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:16:58.798 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:16:58.798 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:16:58.807 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":696,"stamp":"2026-02-12T04:16:57.832319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001333"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73342,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341111+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341111+0000","last_peered":"2026-02-12T04:16:42.341111+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341111+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986400,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:01.809 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:01.989 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:01.989 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:01.999 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":698,"stamp":"2026-02-12T04:17:01.832969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001598"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73342,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341111+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341111+0000","last_peered":"2026-02-12T04:16:42.341111+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341111+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986401,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:02.000 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:02.177 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:02.177 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:02.186 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":698,"stamp":"2026-02-12T04:17:01.832969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001598"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73342,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341111+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341111+0000","last_peered":"2026-02-12T04:16:42.341111+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341111+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986401,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:02.187 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:02.187 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73342, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:42.341111+0000', 'last_change': '2026-02-12T04:16:42.341086+0000', 'last_active': '2026-02-12T04:16:42.341111+0000', 'last_peered': '2026-02-12T04:16:42.341111+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:42.341111+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:02.187 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:02.407 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:02.407 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1345,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:17:02.416 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:02.416 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:02.597 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:02.597 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:02.606 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":698,"stamp":"2026-02-12T04:17:01.832969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001598"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73342,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341111+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341111+0000","last_peered":"2026-02-12T04:16:42.341111+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341111+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986401,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:05.608 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:05.790 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:05.790 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:05.798 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":699,"stamp":"2026-02-12T04:17:03.833189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001601"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73342,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341111+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341111+0000","last_peered":"2026-02-12T04:16:42.341111+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341111+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986401,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:05.799 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:05.975 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:05.976 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:05.984 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":700,"stamp":"2026-02-12T04:17:05.833491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001646"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73342,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341111+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341111+0000","last_peered":"2026-02-12T04:16:42.341111+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341111+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986402,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:05.985 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:05.986 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73342, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:42.341111+0000', 'last_change': '2026-02-12T04:16:42.341086+0000', 'last_active': '2026-02-12T04:16:42.341111+0000', 'last_peered': '2026-02-12T04:16:42.341111+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:42.341111+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:05.986 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:06.255 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:17:06.255+0000 7f4fa981f640 0 --2- 10.20.193.103:0/841108547 >> [v2:10.20.193.103:3300/0,v1:10.20.193.103:6789/0] conn(0x7f4fa411a190 0x7f4fa412aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:17:06.415 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:06.415 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1349,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:17:06.424 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:06.425 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:06.601 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:06.601 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:06.610 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":700,"stamp":"2026-02-12T04:17:05.833491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001646"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73342,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341111+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341111+0000","last_peered":"2026-02-12T04:16:42.341111+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341111+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986402,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:09.612 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:09.791 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:09.791 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:09.800 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":701,"stamp":"2026-02-12T04:17:07.833736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001674"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73342,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341111+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341111+0000","last_peered":"2026-02-12T04:16:42.341111+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341111+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986402,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:09.802 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:09.978 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:09.979 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:09.988 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":702,"stamp":"2026-02-12T04:17:09.834014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001694"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73342,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341111+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341111+0000","last_peered":"2026-02-12T04:16:42.341111+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341111+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986402,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:09.989 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:09.989 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73342, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:42.341111+0000', 'last_change': '2026-02-12T04:16:42.341086+0000', 'last_active': '2026-02-12T04:16:42.341111+0000', 'last_peered': '2026-02-12T04:16:42.341111+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:42.341111+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:09.989 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:10.208 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:10.208 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1353,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:17:10.217 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:10.217 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:10.398 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:10.398 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:10.406 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":702,"stamp":"2026-02-12T04:17:09.834014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001694"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73342,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341111+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341111+0000","last_peered":"2026-02-12T04:16:42.341111+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341111+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986402,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:13.408 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:13.586 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:13.587 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:13.595 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":703,"stamp":"2026-02-12T04:17:11.834249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001587"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73342,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341111+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341111+0000","last_peered":"2026-02-12T04:16:42.341111+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341111+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986403,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:13.596 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:13.772 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:13.772 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:13.781 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":703,"stamp":"2026-02-12T04:17:11.834249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001587"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73342,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:16:42.341111+0000","last_change":"2026-02-12T04:16:42.341086+0000","last_active":"2026-02-12T04:16:42.341111+0000","last_peered":"2026-02-12T04:16:42.341111+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:16:42.341111+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986403,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:13.782 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:13.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73342, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:16:42.341111+0000', 'last_change': '2026-02-12T04:16:42.341086+0000', 'last_active': '2026-02-12T04:16:42.341111+0000', 'last_peered': '2026-02-12T04:16:42.341111+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:16:42.341111+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:13.782 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:14.002 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:14.003 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1357,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:17:14.013 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:14.013 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:14.188 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:14.188 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:14.197 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":704,"stamp":"2026-02-12T04:17:13.834562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986403,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:17.199 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:17.379 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:17.380 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:17.389 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":705,"stamp":"2026-02-12T04:17:15.834822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986404,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:17.390 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:17.567 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:17.568 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:17.576 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":705,"stamp":"2026-02-12T04:17:15.834822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986404,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:17.577 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:17.577 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73346, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:12.341681+0000', 'last_change': '2026-02-12T04:17:12.341681+0000', 'last_active': '2026-02-12T04:17:12.341681+0000', 'last_peered': '2026-02-12T04:17:12.341681+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:12.341681+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:17.578 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:17.800 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:17.800 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1361,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-12T04:15:07.818519+0000","services":{}},"progress_events":{}} 2026-02-12T04:17:17.810 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:17.810 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:17.990 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:17.991 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:18.001 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":706,"stamp":"2026-02-12T04:17:17.835115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001623"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986404,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:21.003 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:21.182 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:21.182 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:21.191 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":707,"stamp":"2026-02-12T04:17:19.835362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001626"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986404,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:21.192 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:21.375 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:21.376 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:21.384 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":707,"stamp":"2026-02-12T04:17:19.835362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001626"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986404,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:21.385 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:21.385 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73346, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:12.341681+0000', 'last_change': '2026-02-12T04:17:12.341681+0000', 'last_active': '2026-02-12T04:17:12.341681+0000', 'last_peered': '2026-02-12T04:17:12.341681+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:12.341681+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:21.386 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:21.606 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:21.607 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1365,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":22,"modified":"2026-02-12T04:17:17.835147+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:17:21.616 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:21.616 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:21.795 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:21.795 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:21.804 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":707,"stamp":"2026-02-12T04:17:19.835362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001626"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986404,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:24.806 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:24.985 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:24.985 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:24.994 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":709,"stamp":"2026-02-12T04:17:23.835952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001702"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986405,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:24.995 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:25.173 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:25.173 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:25.182 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":709,"stamp":"2026-02-12T04:17:23.835952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001702"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986405,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:25.183 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:25.183 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73346, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:12.341681+0000', 'last_change': '2026-02-12T04:17:12.341681+0000', 'last_active': '2026-02-12T04:17:12.341681+0000', 'last_peered': '2026-02-12T04:17:12.341681+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:12.341681+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:25.184 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:25.406 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:25.406 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1368,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":22,"modified":"2026-02-12T04:17:17.835147+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:17:25.415 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:25.416 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:25.594 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:25.594 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:25.603 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":709,"stamp":"2026-02-12T04:17:23.835952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001702"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986405,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:28.605 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:28.784 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:28.785 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:28.794 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":711,"stamp":"2026-02-12T04:17:27.836426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986406,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:28.795 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:28.974 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:28.975 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:28.984 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":711,"stamp":"2026-02-12T04:17:27.836426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986406,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:28.985 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:28.985 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73346, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:12.341681+0000', 'last_change': '2026-02-12T04:17:12.341681+0000', 'last_active': '2026-02-12T04:17:12.341681+0000', 'last_peered': '2026-02-12T04:17:12.341681+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:12.341681+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:28.985 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:29.209 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:29.209 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1372,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":22,"modified":"2026-02-12T04:17:17.835147+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:17:29.218 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:29.218 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:29.394 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:29.395 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:29.403 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":711,"stamp":"2026-02-12T04:17:27.836426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986406,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:32.405 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:32.581 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:32.581 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:32.590 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":713,"stamp":"2026-02-12T04:17:31.837023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001660"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986407,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:32.591 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:32.770 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:32.771 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:32.780 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":713,"stamp":"2026-02-12T04:17:31.837023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001660"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986407,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:32.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:32.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73346, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:12.341681+0000', 'last_change': '2026-02-12T04:17:12.341681+0000', 'last_active': '2026-02-12T04:17:12.341681+0000', 'last_peered': '2026-02-12T04:17:12.341681+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:12.341681+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:32.782 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:33.002 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:33.002 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1376,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-12T04:17:31.837063+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:17:33.013 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:33.013 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:33.194 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:33.194 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:33.203 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":713,"stamp":"2026-02-12T04:17:31.837023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001660"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986407,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:36.204 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:36.389 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:36.389 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:36.398 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2026-02-12T04:17:35.837566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001614"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986408,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:36.399 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:36.576 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:36.576 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:36.585 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2026-02-12T04:17:35.837566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001614"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986408,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:36.586 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:36.586 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73346, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:12.341681+0000', 'last_change': '2026-02-12T04:17:12.341681+0000', 'last_active': '2026-02-12T04:17:12.341681+0000', 'last_peered': '2026-02-12T04:17:12.341681+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:12.341681+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:36.586 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:36.810 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:36.810 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1380,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-12T04:17:31.837063+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:17:36.820 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:36.820 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:36.999 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:36.999 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:37.009 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2026-02-12T04:17:35.837566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001614"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986408,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:40.010 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:40.188 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:40.189 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:40.197 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2026-02-12T04:17:39.838047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001621"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986408,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:40.199 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:40.470 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:40.470 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:40.479 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2026-02-12T04:17:39.838047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001621"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986408,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:40.480 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:40.480 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73346, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:12.341681+0000', 'last_change': '2026-02-12T04:17:12.341681+0000', 'last_active': '2026-02-12T04:17:12.341681+0000', 'last_peered': '2026-02-12T04:17:12.341681+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:12.341681+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:40.481 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:40.704 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:40.704 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1384,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-12T04:17:31.837063+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:17:40.714 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:40.714 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:40.893 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:40.894 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:40.903 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2026-02-12T04:17:39.838047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001621"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73346,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:12.341681+0000","last_change":"2026-02-12T04:17:12.341681+0000","last_active":"2026-02-12T04:17:12.341681+0000","last_peered":"2026-02-12T04:17:12.341681+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:12.341681+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986408,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:43.904 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:44.083 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:44.084 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:44.092 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":719,"stamp":"2026-02-12T04:17:43.838459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001436"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986409,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:44.094 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:44.272 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:44.272 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:44.281 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":719,"stamp":"2026-02-12T04:17:43.838459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001436"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986409,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:44.282 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:44.282 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73350, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:42.342044+0000', 'last_change': '2026-02-12T04:17:42.342044+0000', 'last_active': '2026-02-12T04:17:42.342044+0000', 'last_peered': '2026-02-12T04:17:42.342044+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:42.342044+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:44.283 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:44.503 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:44.503 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1388,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-12T04:17:31.837063+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:17:44.512 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:44.512 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:44.689 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:44.689 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:44.698 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":719,"stamp":"2026-02-12T04:17:43.838459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001436"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986409,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:47.700 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:47.879 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:47.880 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:47.889 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":721,"stamp":"2026-02-12T04:17:47.838879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001312"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986410,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:47.890 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:48.070 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:48.070 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:48.079 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":721,"stamp":"2026-02-12T04:17:47.838879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001312"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986410,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:48.080 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:48.080 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73350, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:42.342044+0000', 'last_change': '2026-02-12T04:17:42.342044+0000', 'last_active': '2026-02-12T04:17:42.342044+0000', 'last_peered': '2026-02-12T04:17:42.342044+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:42.342044+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:48.081 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:48.300 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:48.300 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1391,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-12T04:17:31.837063+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:17:48.309 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:48.310 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:48.485 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:48.486 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:48.494 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":721,"stamp":"2026-02-12T04:17:47.838879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001312"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986410,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:51.496 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:51.675 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:51.675 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:51.684 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":722,"stamp":"2026-02-12T04:17:49.839152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001323"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986410,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:51.685 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:51.861 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:51.861 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:51.869 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":723,"stamp":"2026-02-12T04:17:51.839466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001418"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986411,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:51.870 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:51.871 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73350, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:42.342044+0000', 'last_change': '2026-02-12T04:17:42.342044+0000', 'last_active': '2026-02-12T04:17:42.342044+0000', 'last_peered': '2026-02-12T04:17:42.342044+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:42.342044+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:51.871 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:52.093 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:52.094 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1395,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-12T04:17:31.837063+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:17:52.104 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:52.104 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:52.284 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:52.284 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:52.293 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":723,"stamp":"2026-02-12T04:17:51.839466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001418"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986411,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:55.295 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:55.476 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:55.477 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:55.486 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":724,"stamp":"2026-02-12T04:17:53.839713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001451"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986411,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:55.487 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:55.666 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:55.666 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:55.675 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":724,"stamp":"2026-02-12T04:17:53.839713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001451"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986411,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:55.676 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:55.676 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73350, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:42.342044+0000', 'last_change': '2026-02-12T04:17:42.342044+0000', 'last_active': '2026-02-12T04:17:42.342044+0000', 'last_peered': '2026-02-12T04:17:42.342044+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:42.342044+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:55.677 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:55.908 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:55.908 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1399,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-12T04:17:31.837063+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:17:55.917 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:55.918 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:56.096 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:56.096 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:56.105 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":725,"stamp":"2026-02-12T04:17:55.840000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001540"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986412,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:59.106 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:59.291 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:59.291 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:59.300 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":726,"stamp":"2026-02-12T04:17:57.840243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001568"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986412,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:59.301 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:59.482 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:59.482 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:59.491 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":726,"stamp":"2026-02-12T04:17:57.840243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001568"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986412,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:17:59.492 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:17:59.492 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73350, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:42.342044+0000', 'last_change': '2026-02-12T04:17:42.342044+0000', 'last_active': '2026-02-12T04:17:42.342044+0000', 'last_peered': '2026-02-12T04:17:42.342044+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:42.342044+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:17:59.493 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:17:59.712 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:59.713 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1403,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-12T04:17:31.837063+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:17:59.722 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:17:59.723 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:17:59.902 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:17:59.903 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:17:59.911 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":727,"stamp":"2026-02-12T04:17:59.840536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001656"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986412,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:02.914 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:03.092 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:03.092 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:03.101 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":728,"stamp":"2026-02-12T04:18:01.840775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986413,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:03.102 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:03.279 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:03.280 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:03.288 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":728,"stamp":"2026-02-12T04:18:01.840775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986413,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:03.289 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:03.289 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73350, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:42.342044+0000', 'last_change': '2026-02-12T04:17:42.342044+0000', 'last_active': '2026-02-12T04:17:42.342044+0000', 'last_peered': '2026-02-12T04:17:42.342044+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:42.342044+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:03.289 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:03.509 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:03.510 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1407,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-12T04:17:31.837063+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:03.519 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:03.519 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:03.696 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:03.696 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:03.705 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":728,"stamp":"2026-02-12T04:18:01.840775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986413,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:06.707 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:06.884 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:06.885 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:06.894 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":730,"stamp":"2026-02-12T04:18:05.841268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001554"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986414,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:06.895 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:07.073 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:07.073 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:07.082 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":730,"stamp":"2026-02-12T04:18:05.841268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001554"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986414,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:07.083 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:07.083 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73350, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:42.342044+0000', 'last_change': '2026-02-12T04:17:42.342044+0000', 'last_active': '2026-02-12T04:17:42.342044+0000', 'last_peered': '2026-02-12T04:17:42.342044+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:42.342044+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:07.083 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:07.305 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:07.306 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1410,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-12T04:17:31.837063+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:07.315 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:07.315 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:07.496 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:07.497 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:07.505 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":730,"stamp":"2026-02-12T04:18:05.841268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001554"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986414,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:10.507 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:10.685 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:10.685 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:10.695 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":732,"stamp":"2026-02-12T04:18:09.841753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001510"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986414,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:10.696 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:10.871 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:10.871 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:10.880 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":732,"stamp":"2026-02-12T04:18:09.841753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001510"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986414,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:10.881 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:10.881 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73350, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:17:42.342044+0000', 'last_change': '2026-02-12T04:17:42.342044+0000', 'last_active': '2026-02-12T04:17:42.342044+0000', 'last_peered': '2026-02-12T04:17:42.342044+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:17:42.342044+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:10.881 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:11.102 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:11.102 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1414,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":24,"modified":"2026-02-12T04:18:07.841554+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:11.112 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:11.112 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:11.287 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:11.288 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:11.296 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":732,"stamp":"2026-02-12T04:18:09.841753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001510"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73350,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:17:42.342044+0000","last_change":"2026-02-12T04:17:42.342044+0000","last_active":"2026-02-12T04:17:42.342044+0000","last_peered":"2026-02-12T04:17:42.342044+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:17:42.342044+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986414,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:14.299 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:14.480 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:14.481 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:14.491 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":734,"stamp":"2026-02-12T04:18:13.842211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001436"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986415,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:14.492 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:14.668 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:14.669 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:14.677 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":734,"stamp":"2026-02-12T04:18:13.842211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001436"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986415,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:14.678 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:14.679 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73354, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:18:12.342627+0000', 'last_change': '2026-02-12T04:18:12.342627+0000', 'last_active': '2026-02-12T04:18:12.342627+0000', 'last_peered': '2026-02-12T04:18:12.342627+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:18:12.342627+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:14.679 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:14.899 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:14.899 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1418,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":24,"modified":"2026-02-12T04:18:07.841554+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:14.909 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:14.909 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:15.095 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:15.095 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:15.103 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":734,"stamp":"2026-02-12T04:18:13.842211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001436"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986415,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:18.105 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:18.285 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:18.286 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:18.295 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":736,"stamp":"2026-02-12T04:18:17.842693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001424"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986416,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:18.296 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:18.476 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:18.476 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:18.485 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":736,"stamp":"2026-02-12T04:18:17.842693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001424"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986416,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:18.486 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:18.486 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73354, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:18:12.342627+0000', 'last_change': '2026-02-12T04:18:12.342627+0000', 'last_active': '2026-02-12T04:18:12.342627+0000', 'last_peered': '2026-02-12T04:18:12.342627+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:18:12.342627+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:18.487 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:18.712 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:18.712 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1422,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":24,"modified":"2026-02-12T04:18:07.841554+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:18.722 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:18.722 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:18.901 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:18.902 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:18.911 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":736,"stamp":"2026-02-12T04:18:17.842693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001424"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986416,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:21.913 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:22.092 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:22.092 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:22.101 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":738,"stamp":"2026-02-12T04:18:21.843151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001398"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986417,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:22.102 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:22.280 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:22.281 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:22.289 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":738,"stamp":"2026-02-12T04:18:21.843151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001398"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986417,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:22.290 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:22.290 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73354, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:18:12.342627+0000', 'last_change': '2026-02-12T04:18:12.342627+0000', 'last_active': '2026-02-12T04:18:12.342627+0000', 'last_peered': '2026-02-12T04:18:12.342627+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:18:12.342627+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:22.291 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:22.512 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:22.512 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1426,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":24,"modified":"2026-02-12T04:18:07.841554+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:22.521 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:22.522 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:22.698 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:22.699 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:22.707 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":738,"stamp":"2026-02-12T04:18:21.843151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001398"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986417,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:25.709 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:25.890 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:25.891 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:25.900 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":740,"stamp":"2026-02-12T04:18:25.843752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001540"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986418,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:25.901 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:26.079 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:26.092 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:26.093 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":740,"stamp":"2026-02-12T04:18:25.843752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001540"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986418,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:26.094 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:26.094 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73354, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:18:12.342627+0000', 'last_change': '2026-02-12T04:18:12.342627+0000', 'last_active': '2026-02-12T04:18:12.342627+0000', 'last_peered': '2026-02-12T04:18:12.342627+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:18:12.342627+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:26.094 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:26.315 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:26.315 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1429,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":24,"modified":"2026-02-12T04:18:07.841554+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:26.325 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:26.326 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:26.503 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:26.503 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:26.512 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":740,"stamp":"2026-02-12T04:18:25.843752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001540"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986418,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:29.515 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:29.696 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:29.696 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:29.705 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":741,"stamp":"2026-02-12T04:18:27.843997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001557"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986418,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:29.706 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:29.887 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:29.887 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:29.896 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":742,"stamp":"2026-02-12T04:18:29.844293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001600"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986418,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:29.897 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:29.897 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73354, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:18:12.342627+0000', 'last_change': '2026-02-12T04:18:12.342627+0000', 'last_active': '2026-02-12T04:18:12.342627+0000', 'last_peered': '2026-02-12T04:18:12.342627+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:18:12.342627+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:29.898 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:30.119 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:30.120 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1433,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":24,"modified":"2026-02-12T04:18:07.841554+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:30.130 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:30.130 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:30.307 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:30.307 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:30.316 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":742,"stamp":"2026-02-12T04:18:29.844293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001600"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986418,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:33.318 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:33.590 INFO:teuthology.orchestra.run.trial103.stderr:2026-02-12T04:18:33.588+0000 7fab4dd6e640 0 --2- 10.20.193.103:0/2237667259 >> [v2:10.20.193.103:3301/0,v1:10.20.193.103:6790/0] conn(0x7fab48154e40 0x7fab48175220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-12T04:18:33.698 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:33.698 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:33.707 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":743,"stamp":"2026-02-12T04:18:31.844546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986419,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:33.708 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:33.888 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:33.889 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:33.898 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":744,"stamp":"2026-02-12T04:18:33.844858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001706"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986419,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:33.899 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:33.899 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73354, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:18:12.342627+0000', 'last_change': '2026-02-12T04:18:12.342627+0000', 'last_active': '2026-02-12T04:18:12.342627+0000', 'last_peered': '2026-02-12T04:18:12.342627+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:18:12.342627+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:33.899 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:34.122 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:34.122 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1437,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":24,"modified":"2026-02-12T04:18:07.841554+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:34.132 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:34.132 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:34.309 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:34.310 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:34.319 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":744,"stamp":"2026-02-12T04:18:33.844858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001706"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986419,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:37.321 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:37.500 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:37.501 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:37.509 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":745,"stamp":"2026-02-12T04:18:35.845142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001698"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986420,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:37.510 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:37.692 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:37.692 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:37.700 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":745,"stamp":"2026-02-12T04:18:35.845142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001698"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986420,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:37.701 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:37.701 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73354, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:18:12.342627+0000', 'last_change': '2026-02-12T04:18:12.342627+0000', 'last_active': '2026-02-12T04:18:12.342627+0000', 'last_peered': '2026-02-12T04:18:12.342627+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:18:12.342627+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:37.702 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:37.924 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:37.925 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1441,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":24,"modified":"2026-02-12T04:18:07.841554+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:37.935 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:37.935 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:38.113 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:38.113 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:38.122 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2026-02-12T04:18:37.845427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001675"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986420,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:41.124 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:41.303 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:41.304 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:41.313 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":747,"stamp":"2026-02-12T04:18:39.845689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001691"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986420,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:41.314 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:41.490 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:41.491 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:41.500 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":747,"stamp":"2026-02-12T04:18:39.845689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001691"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986420,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:41.501 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:41.501 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73354, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:18:12.342627+0000', 'last_change': '2026-02-12T04:18:12.342627+0000', 'last_active': '2026-02-12T04:18:12.342627+0000', 'last_peered': '2026-02-12T04:18:12.342627+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:18:12.342627+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:41.501 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:41.723 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:41.723 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1445,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":25,"modified":"2026-02-12T04:18:37.845459+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:41.733 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:41.733 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:41.917 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:41.918 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:41.926 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":748,"stamp":"2026-02-12T04:18:41.845986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001692"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73354,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:12.342627+0000","last_change":"2026-02-12T04:18:12.342627+0000","last_active":"2026-02-12T04:18:12.342627+0000","last_peered":"2026-02-12T04:18:12.342627+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:12.342627+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986421,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:44.928 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:45.109 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:45.109 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:45.119 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":749,"stamp":"2026-02-12T04:18:43.846203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001657"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73358,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:42.343054+0000","last_change":"2026-02-12T04:18:42.343054+0000","last_active":"2026-02-12T04:18:42.343054+0000","last_peered":"2026-02-12T04:18:42.343054+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:42.343054+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986421,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:45.120 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:45.298 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:45.299 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:45.307 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":749,"stamp":"2026-02-12T04:18:43.846203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001657"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73358,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:42.343054+0000","last_change":"2026-02-12T04:18:42.343054+0000","last_active":"2026-02-12T04:18:42.343054+0000","last_peered":"2026-02-12T04:18:42.343054+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:42.343054+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986421,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:45.308 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:45.308 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73358, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:18:42.343054+0000', 'last_change': '2026-02-12T04:18:42.343054+0000', 'last_active': '2026-02-12T04:18:42.343054+0000', 'last_peered': '2026-02-12T04:18:42.343054+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:18:42.343054+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:45.309 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:45.534 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:45.534 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1449,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":25,"modified":"2026-02-12T04:18:37.845459+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:45.544 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:45.544 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:45.724 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:45.724 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:45.733 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":749,"stamp":"2026-02-12T04:18:43.846203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001657"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73358,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:42.343054+0000","last_change":"2026-02-12T04:18:42.343054+0000","last_active":"2026-02-12T04:18:42.343054+0000","last_peered":"2026-02-12T04:18:42.343054+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:42.343054+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:14:40.287864+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:14:40.287864+0000","last_peered":"2026-02-12T04:14:40.287864+0000","last_clean":"2026-02-12T04:14:40.287864+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:14:40.287864+0000","last_undegraded":"2026-02-12T04:14:40.287864+0000","last_fullsized":"2026-02-12T04:14:40.287864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986421,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:48.735 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:48.919 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:48.920 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:48.929 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":751,"stamp":"2026-02-12T04:18:47.846744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001602"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73358,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:42.343054+0000","last_change":"2026-02-12T04:18:42.343054+0000","last_active":"2026-02-12T04:18:42.343054+0000","last_peered":"2026-02-12T04:18:42.343054+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:42.343054+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:16:45.223569+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:16:45.223569+0000","last_peered":"2026-02-12T04:16:45.223569+0000","last_clean":"2026-02-12T04:16:45.223569+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:16:45.223569+0000","last_undegraded":"2026-02-12T04:16:45.223569+0000","last_fullsized":"2026-02-12T04:16:45.223569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986422,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:48.930 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:49.107 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:49.107 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:49.117 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":751,"stamp":"2026-02-12T04:18:47.846744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001602"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73358,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:42.343054+0000","last_change":"2026-02-12T04:18:42.343054+0000","last_active":"2026-02-12T04:18:42.343054+0000","last_peered":"2026-02-12T04:18:42.343054+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:42.343054+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:16:45.223569+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:16:45.223569+0000","last_peered":"2026-02-12T04:16:45.223569+0000","last_clean":"2026-02-12T04:16:45.223569+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:16:45.223569+0000","last_undegraded":"2026-02-12T04:16:45.223569+0000","last_fullsized":"2026-02-12T04:16:45.223569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986422,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:49.117 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:49.117 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73358, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:18:42.343054+0000', 'last_change': '2026-02-12T04:18:42.343054+0000', 'last_active': '2026-02-12T04:18:42.343054+0000', 'last_peered': '2026-02-12T04:18:42.343054+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:18:42.343054+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:49.118 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:49.337 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:49.337 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1452,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":25,"modified":"2026-02-12T04:18:37.845459+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:49.347 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:49.347 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:49.527 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:49.527 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:49.536 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":751,"stamp":"2026-02-12T04:18:47.846744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001602"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73358,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:42.343054+0000","last_change":"2026-02-12T04:18:42.343054+0000","last_active":"2026-02-12T04:18:42.343054+0000","last_peered":"2026-02-12T04:18:42.343054+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:42.343054+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:16:45.223569+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:16:45.223569+0000","last_peered":"2026-02-12T04:16:45.223569+0000","last_clean":"2026-02-12T04:16:45.223569+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:16:45.223569+0000","last_undegraded":"2026-02-12T04:16:45.223569+0000","last_fullsized":"2026-02-12T04:16:45.223569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986422,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:52.538 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:52.717 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:52.717 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:52.726 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":753,"stamp":"2026-02-12T04:18:51.847270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001580"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73358,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:42.343054+0000","last_change":"2026-02-12T04:18:42.343054+0000","last_active":"2026-02-12T04:18:42.343054+0000","last_peered":"2026-02-12T04:18:42.343054+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:42.343054+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:16:45.223569+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:16:45.223569+0000","last_peered":"2026-02-12T04:16:45.223569+0000","last_clean":"2026-02-12T04:16:45.223569+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:16:45.223569+0000","last_undegraded":"2026-02-12T04:16:45.223569+0000","last_fullsized":"2026-02-12T04:16:45.223569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986423,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:52.727 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:52.904 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:52.905 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:52.914 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":753,"stamp":"2026-02-12T04:18:51.847270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001580"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73358,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:42.343054+0000","last_change":"2026-02-12T04:18:42.343054+0000","last_active":"2026-02-12T04:18:42.343054+0000","last_peered":"2026-02-12T04:18:42.343054+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:42.343054+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:16:45.223569+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:16:45.223569+0000","last_peered":"2026-02-12T04:16:45.223569+0000","last_clean":"2026-02-12T04:16:45.223569+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:16:45.223569+0000","last_undegraded":"2026-02-12T04:16:45.223569+0000","last_fullsized":"2026-02-12T04:16:45.223569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986423,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:52.915 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:52.915 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73358, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:18:42.343054+0000', 'last_change': '2026-02-12T04:18:42.343054+0000', 'last_active': '2026-02-12T04:18:42.343054+0000', 'last_peered': '2026-02-12T04:18:42.343054+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:18:42.343054+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:52.915 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:53.134 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:53.135 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1456,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":25,"modified":"2026-02-12T04:18:37.845459+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:53.144 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:53.144 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:53.320 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:53.321 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:53.328 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":753,"stamp":"2026-02-12T04:18:51.847270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001580"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73358,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:42.343054+0000","last_change":"2026-02-12T04:18:42.343054+0000","last_active":"2026-02-12T04:18:42.343054+0000","last_peered":"2026-02-12T04:18:42.343054+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:42.343054+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:16:45.223569+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:16:45.223569+0000","last_peered":"2026-02-12T04:16:45.223569+0000","last_clean":"2026-02-12T04:16:45.223569+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:16:45.223569+0000","last_undegraded":"2026-02-12T04:16:45.223569+0000","last_fullsized":"2026-02-12T04:16:45.223569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986423,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:56.330 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:56.607 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:56.607 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:56.617 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":755,"stamp":"2026-02-12T04:18:55.847905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73358,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:42.343054+0000","last_change":"2026-02-12T04:18:42.343054+0000","last_active":"2026-02-12T04:18:42.343054+0000","last_peered":"2026-02-12T04:18:42.343054+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:42.343054+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:16:45.223569+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:16:45.223569+0000","last_peered":"2026-02-12T04:16:45.223569+0000","last_clean":"2026-02-12T04:16:45.223569+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:16:45.223569+0000","last_undegraded":"2026-02-12T04:16:45.223569+0000","last_fullsized":"2026-02-12T04:16:45.223569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986424,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:56.618 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:56.798 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:56.798 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:56.808 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":755,"stamp":"2026-02-12T04:18:55.847905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73358,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:42.343054+0000","last_change":"2026-02-12T04:18:42.343054+0000","last_active":"2026-02-12T04:18:42.343054+0000","last_peered":"2026-02-12T04:18:42.343054+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:42.343054+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:16:45.223569+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:16:45.223569+0000","last_peered":"2026-02-12T04:16:45.223569+0000","last_clean":"2026-02-12T04:16:45.223569+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:16:45.223569+0000","last_undegraded":"2026-02-12T04:16:45.223569+0000","last_fullsized":"2026-02-12T04:16:45.223569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986424,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:56.809 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:56.809 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73358, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:18:42.343054+0000', 'last_change': '2026-02-12T04:18:42.343054+0000', 'last_active': '2026-02-12T04:18:42.343054+0000', 'last_peered': '2026-02-12T04:18:42.343054+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:18:42.343054+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:56.809 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-12T04:18:57.030 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:57.030 INFO:teuthology.orchestra.run.trial103.stdout:{"fsid":"e9633232-3262-4ad1-a60a-815554f29d9f","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18335/55009 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1460,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1770868721,"num_in_osds":4,"osd_in_since":1770868478,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":18337,"data_bytes":76898697752,"bytes_used":135745949696,"bytes_avail":250801106944,"bytes_total":386547056640,"degraded_objects":18335,"degraded_total":55009,"degraded_ratio":0.33330909487538402},"fsmap":{"epoch":1,"btime":"2026-02-12T03:54:36:498858+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":25,"modified":"2026-02-12T04:18:37.845459+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-12T04:18:57.039 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-12T04:18:57.039 INFO:tasks.ceph.ceph_manager.ceph:dumping pgs not clean 2026-02-12T04:18:57.040 DEBUG:teuthology.orchestra.run.trial103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T04:18:57.224 INFO:teuthology.orchestra.run.trial103.stdout: 2026-02-12T04:18:57.225 INFO:teuthology.orchestra.run.trial103.stderr:dumped all 2026-02-12T04:18:57.234 INFO:teuthology.orchestra.run.trial103.stdout:{"pg_ready":true,"pg_map":{"version":755,"stamp":"2026-02-12T04:18:55.847905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76898697752,"num_objects":18337,"num_object_clones":0,"num_object_copies":55009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18337,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":73176,"num_write_kb":187330884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":132564404,"kb_used_data":131215092,"kb_used_omap":62,"kb_used_meta":1349185,"kb_avail":244922956,"statfs":{"total":386547056640,"available":250801106944,"internally_reserved":0,"allocated":134364254208,"data_stored":134363516640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63958,"internal_metadata":1381565994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"25'73139","reported_seq":73358,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-12T04:18:42.343054+0000","last_change":"2026-02-12T04:18:42.343054+0000","last_active":"2026-02-12T04:18:42.343054+0000","last_peered":"2026-02-12T04:18:42.343054+0000","last_clean":"2026-02-12T03:57:03.665734+0000","last_became_active":"2026-02-12T03:58:42.321232+0000","last_became_peered":"2026-02-12T03:58:42.321232+0000","last_unstale":"2026-02-12T04:18:42.343054+0000","last_undegraded":"2026-02-12T03:58:42.311176+0000","last_fullsized":"2026-02-12T03:58:42.309791+0000","mapping_epoch":32,"log_start":"25'73132","ondisk_log_start":"25'73132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:51.686565+0000","last_clean_scrub_stamp":"2026-02-12T03:54:51.686565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":3000,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":18335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-12T04:16:45.223569+0000","last_change":"2026-02-12T03:58:41.309386+0000","last_active":"2026-02-12T04:16:45.223569+0000","last_peered":"2026-02-12T04:16:45.223569+0000","last_clean":"2026-02-12T04:16:45.223569+0000","last_became_active":"2026-02-12T03:58:41.309265+0000","last_became_peered":"2026-02-12T03:58:41.309265+0000","last_unstale":"2026-02-12T04:16:45.223569+0000","last_undegraded":"2026-02-12T04:16:45.223569+0000","last_fullsized":"2026-02-12T04:16:45.223569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T03:54:44.663759+0000","last_clean_scrub_stamp":"2026-02-12T03:54:44.663759+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-13T10:47:57.894308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":76898369560,"num_objects":18335,"num_object_clones":0,"num_object_copies":55005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73139,"num_write_kb":187330562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134362443776,"data_stored":134362443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":20,"kb_used_meta":640235,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20502,"internal_metadata":655601642},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38188588,"kb_used_data":37548316,"kb_used_omap":14,"kb_used_meta":640241,"kb_avail":56183252,"statfs":{"total":96636764160,"available":57531650048,"internally_reserved":0,"allocated":38449475584,"data_stored":38449292984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15296,"internal_metadata":655606848},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":31,"seq":133143986424,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56159664,"kb_used_data":56117856,"kb_used_omap":7,"kb_used_meta":41784,"kb_avail":38212176,"statfs":{"total":96636764160,"available":39129268224,"internally_reserved":0,"allocated":57464684544,"data_stored":57464498360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7656,"internal_metadata":42787352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":604,"kb_used_omap":20,"kb_used_meta":26923,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":618496,"data_stored":432312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20504,"internal_metadata":27570152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57464066048,"data_stored":57464066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38449188864,"data_stored":38449188864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T04:18:57.235 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-12T04:18:57.235 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "25'73139", 'reported_seq': 73358, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-12T04:18:42.343054+0000', 'last_change': '2026-02-12T04:18:42.343054+0000', 'last_active': '2026-02-12T04:18:42.343054+0000', 'last_peered': '2026-02-12T04:18:42.343054+0000', 'last_clean': '2026-02-12T03:57:03.665734+0000', 'last_became_active': '2026-02-12T03:58:42.321232+0000', 'last_became_peered': '2026-02-12T03:58:42.321232+0000', 'last_unstale': '2026-02-12T04:18:42.343054+0000', 'last_undegraded': '2026-02-12T03:58:42.311176+0000', 'last_fullsized': '2026-02-12T03:58:42.309791+0000', 'mapping_epoch': 32, 'log_start': "25'73132", 'ondisk_log_start': "25'73132", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'last_clean_scrub_stamp': '2026-02-12T03:54:51.686565+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 3000, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 76898369560, 'num_objects': 18335, 'num_object_clones': 0, 'num_object_copies': 55005, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 18335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 18335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 73139, 'num_write_kb': 187330562, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 18335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-12T04:18:57.236 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f66ecfb34c0370410e78b3ee641753d19da653b/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_1279775fef08a9169052c6dd6a10581353958aac/qa/tasks/ceph.py", line 2011, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_1279775fef08a9169052c6dd6a10581353958aac/qa/tasks/ceph_manager.py", line 2930, in wait_for_clean assert time.time() - start < timeout, \ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: wait_for_clean: failed before timeout expired 2026-02-12T04:18:57.238 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-12T04:18:57.238 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-12T04:18:57.239 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-12T04:18:57.239 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T04:18:57.272 INFO:tasks.ceph.osd.0:Stopped 2026-02-12T04:18:57.272 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-12T04:18:57.272 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T04:18:57.347 INFO:tasks.ceph.osd.1:Stopped 2026-02-12T04:18:57.347 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-12T04:18:57.347 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T04:18:57.436 INFO:tasks.ceph.osd.2:Stopped 2026-02-12T04:18:57.436 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-12T04:18:57.437 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T04:18:57.490 INFO:tasks.ceph.osd.3:Stopped 2026-02-12T04:18:57.490 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-12T04:18:57.490 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-12T04:18:57.491 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T04:18:57.521 INFO:tasks.ceph.mgr.x:Stopped 2026-02-12T04:18:57.521 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-12T04:18:57.521 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-12T04:18:57.521 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T04:18:57.572 INFO:tasks.ceph.mon.a:Stopped 2026-02-12T04:18:57.573 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2026-02-12T04:18:57.573 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T04:18:57.622 INFO:tasks.ceph.mon.b:Stopped 2026-02-12T04:18:57.622 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2026-02-12T04:18:57.622 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T04:18:57.672 INFO:tasks.ceph.mon.c:Stopped 2026-02-12T04:18:57.673 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-12T04:18:57.673 DEBUG:teuthology.orchestra.run.trial103:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v Error | egrep -v 'overall HEALTH_' | egrep -v '\(OBJECT_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_BACKFILLFULL\)' | egrep -v '\(POOL_NEARFULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'Monitor daemon marked osd\.[[:digit:]]+ down, but it is still running' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2026-02-12T04:18:57.740 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial103.front.sepia.ceph.com 2026-02-12T04:18:57.740 DEBUG:teuthology.orchestra.run.trial103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-12T04:18:57.811 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial103.front.sepia.ceph.com 2026-02-12T04:18:57.811 DEBUG:teuthology.orchestra.run.trial103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-12T04:18:57.897 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial103.front.sepia.ceph.com 2026-02-12T04:18:57.898 DEBUG:teuthology.orchestra.run.trial103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-12T04:18:57.961 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@trial103.front.sepia.ceph.com 2026-02-12T04:18:57.961 DEBUG:teuthology.orchestra.run.trial103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2026-02-12T04:18:58.026 INFO:tasks.ceph:Archiving mon data... 2026-02-12T04:18:58.028 DEBUG:teuthology.misc:Transferring archived files from trial103:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-11_20:27:55-rados-wip-yuri3-testing-2026-02-09-1550-tentacle-distro-default-trial/45778/data/mon.a.tgz 2026-02-12T04:18:58.028 DEBUG:teuthology.orchestra.run.trial103:> mktemp 2026-02-12T04:18:58.044 INFO:teuthology.orchestra.run.trial103.stdout:/tmp/tmp.v8TVFMSudf 2026-02-12T04:18:58.045 DEBUG:teuthology.orchestra.run.trial103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.v8TVFMSudf 2026-02-12T04:18:58.151 DEBUG:teuthology.orchestra.run.trial103:> sudo chmod 0666 /tmp/tmp.v8TVFMSudf 2026-02-12T04:18:58.232 DEBUG:teuthology.orchestra.remote:trial103:/tmp/tmp.v8TVFMSudf is 360KB 2026-02-12T04:18:58.327 DEBUG:teuthology.orchestra.run.trial103:> rm -fr /tmp/tmp.v8TVFMSudf 2026-02-12T04:18:58.344 DEBUG:teuthology.misc:Transferring archived files from trial103:/var/lib/ceph/mon/ceph-b to /home/teuthworker/mnt/teuthology/yuriw-2026-02-11_20:27:55-rados-wip-yuri3-testing-2026-02-09-1550-tentacle-distro-default-trial/45778/data/mon.b.tgz 2026-02-12T04:18:58.345 DEBUG:teuthology.orchestra.run.trial103:> mktemp 2026-02-12T04:18:58.401 INFO:teuthology.orchestra.run.trial103.stdout:/tmp/tmp.tR8ne9cZqw 2026-02-12T04:18:58.401 DEBUG:teuthology.orchestra.run.trial103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.tR8ne9cZqw 2026-02-12T04:18:58.504 DEBUG:teuthology.orchestra.run.trial103:> sudo chmod 0666 /tmp/tmp.tR8ne9cZqw 2026-02-12T04:18:58.585 DEBUG:teuthology.orchestra.remote:trial103:/tmp/tmp.tR8ne9cZqw is 364KB 2026-02-12T04:18:58.679 DEBUG:teuthology.orchestra.run.trial103:> rm -fr /tmp/tmp.tR8ne9cZqw 2026-02-12T04:18:58.696 DEBUG:teuthology.misc:Transferring archived files from trial103:/var/lib/ceph/mon/ceph-c to /home/teuthworker/mnt/teuthology/yuriw-2026-02-11_20:27:55-rados-wip-yuri3-testing-2026-02-09-1550-tentacle-distro-default-trial/45778/data/mon.c.tgz 2026-02-12T04:18:58.696 DEBUG:teuthology.orchestra.run.trial103:> mktemp 2026-02-12T04:18:58.751 INFO:teuthology.orchestra.run.trial103.stdout:/tmp/tmp.9eGNp8reYt 2026-02-12T04:18:58.752 DEBUG:teuthology.orchestra.run.trial103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.9eGNp8reYt 2026-02-12T04:18:58.856 DEBUG:teuthology.orchestra.run.trial103:> sudo chmod 0666 /tmp/tmp.9eGNp8reYt 2026-02-12T04:18:58.938 DEBUG:teuthology.orchestra.remote:trial103:/tmp/tmp.9eGNp8reYt is 364KB 2026-02-12T04:18:59.033 DEBUG:teuthology.orchestra.run.trial103:> rm -fr /tmp/tmp.9eGNp8reYt 2026-02-12T04:18:59.049 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-12T04:18:59.050 DEBUG:teuthology.orchestra.run.trial103:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-12T04:18:59.171 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-12T04:18:59.171 INFO:tasks.ceph:Archiving crash dumps... 2026-02-12T04:18:59.173 DEBUG:teuthology.misc:Transferring archived files from trial103:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-11_20:27:55-rados-wip-yuri3-testing-2026-02-09-1550-tentacle-distro-default-trial/45778/remote/trial103/crash 2026-02-12T04:18:59.174 DEBUG:teuthology.orchestra.run.trial103:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-12T04:18:59.235 INFO:tasks.ceph:Compressing logs... 2026-02-12T04:18:59.235 DEBUG:teuthology.orchestra.run.trial103:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-12T04:18:59.299 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46815.log 2026-02-12T04:18:59.331 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73544.log 2026-02-12T04:18:59.331 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65212.log 2026-02-12T04:18:59.332 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46815.log.gz 2026-02-12T04:18:59.332 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73544.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.73544.log.gz -5 2026-02-12T04:18:59.332 INFO:teuthology.orchestra.run.trial103.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41212.log 2026-02-12T04:18:59.332 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65212.log.gz 2026-02-12T04:18:59.332 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34614.log 2026-02-12T04:18:59.332 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41212.log.gz 2026-02-12T04:18:59.332 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83165.log 2026-02-12T04:18:59.332 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-12T04:18:59.332 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.34614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34614.log.gz 2026-02-12T04:18:59.332 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83165.log.gz 2026-02-12T04:18:59.332 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84978.log 2026-02-12T04:18:59.332 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30427.log 2026-02-12T04:18:59.333 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83753.log 2026-02-12T04:18:59.333 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84978.log.gz 2026-02-12T04:18:59.333 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88457.log 2026-02-12T04:18:59.333 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30427.log.gz 2026-02-12T04:18:59.333 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83753.log.gz 2026-02-12T04:18:59.333 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75455.log 2026-02-12T04:18:59.333 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44756.log 2026-02-12T04:18:59.333 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88457.log.gz 2026-02-12T04:18:59.333 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64182.log 2026-02-12T04:18:59.333 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75455.log.gz 2026-02-12T04:18:59.333 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74573.log 2026-02-12T04:18:59.333 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44756.log.gz 2026-02-12T04:18:59.333 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64182.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.64182.log.gz -5 2026-02-12T04:18:59.333 INFO:teuthology.orchestra.run.trial103.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53188.log 2026-02-12T04:18:59.334 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59525.log 2026-02-12T04:18:59.334 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74573.log.gz 2026-02-12T04:18:59.334 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53188.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53188.log.gz 2026-02-12T04:18:59.334 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40624.log 2026-02-12T04:18:59.334 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59525.log.gz 2026-02-12T04:18:59.334 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67809.log 2026-02-12T04:18:59.334 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40624.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40624.log.gz -- 2026-02-12T04:18:59.334 INFO:teuthology.orchestra.run.trial103.stderr: /var/log/ceph/ceph-client.admin.90172.log 2026-02-12T04:18:59.334 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44315.log 2026-02-12T04:18:59.334 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67809.log.gz 2026-02-12T04:18:59.334 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87085.log 2026-02-12T04:18:59.334 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90172.log.gz 2026-02-12T04:18:59.334 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78801.log 2026-02-12T04:18:59.334 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44315.log.gz 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76827.log 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87085.log.gz 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78801.log.gz 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82577.log 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr:gzip/var/log/ceph/ceph-client.admin.76827.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48876.log 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76827.log.gz 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77023.log 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82577.log.gz 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48876.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48876.log.gz 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36408.log 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68838.log 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.36408.log: -- /var/log/ceph/ceph-client.admin.30826.log 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36408.log.gz 2026-02-12T04:18:59.335 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63496.log 0.0% 2026-02-12T04:18:59.336 INFO:teuthology.orchestra.run.trial103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68838.log.gz 2026-02-12T04:18:59.336 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77023.log.gz 2026-02-12T04:18:59.336 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82479.log 2026-02-12T04:18:59.336 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30826.log.gz 2026-02-12T04:18:59.336 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63496.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.63496.log.gz -5 2026-02-12T04:18:59.336 INFO:teuthology.orchestra.run.trial103.stderr: --verbose -- /var/log/ceph/ceph-client.admin.75161.log 2026-02-12T04:18:59.336 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82479.log.gz 2026-02-12T04:18:59.336 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50052.log 2026-02-12T04:18:59.336 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73691.log 2026-02-12T04:18:59.336 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75161.log.gz 2026-02-12T04:18:59.336 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50052.log.gz 2026-02-12T04:18:59.336 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72613.log 2026-02-12T04:18:59.336 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80764.log 2026-02-12T04:18:59.336 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73691.log.gz 2026-02-12T04:18:59.337 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68054.log 2026-02-12T04:18:59.337 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72613.log.gz 2026-02-12T04:18:59.337 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77464.log 2026-02-12T04:18:59.337 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80764.log.gz 2026-02-12T04:18:59.337 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68054.log.gz 2026-02-12T04:18:59.337 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41849.log 2026-02-12T04:18:59.337 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64722.log 0.0% 2026-02-12T04:18:59.337 INFO:teuthology.orchestra.run.trial103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77464.log.gz 2026-02-12T04:18:59.337 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75700.log 2026-02-12T04:18:59.337 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41849.log.gz 2026-02-12T04:18:59.337 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64722.log.gz 2026-02-12T04:18:59.337 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67319.log 2026-02-12T04:18:59.337 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82038.log 2026-02-12T04:18:59.337 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75700.log.gz 2026-02-12T04:18:59.338 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91103.log 2026-02-12T04:18:59.338 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67319.log.gz 2026-02-12T04:18:59.338 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82038.log.gz 2026-02-12T04:18:59.338 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62810.log 2026-02-12T04:18:59.338 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70359.log 2026-02-12T04:18:59.338 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91103.log.gz 2026-02-12T04:18:59.338 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62810.log.gz 2026-02-12T04:18:59.338 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83557.log 2026-02-12T04:18:59.338 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44805.log 2026-02-12T04:18:59.338 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70359.log.gz 2026-02-12T04:18:59.338 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83557.log.gz 2026-02-12T04:18:59.338 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78950.log 2026-02-12T04:18:59.338 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44805.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44805.log.gz -5 2026-02-12T04:18:59.338 INFO:teuthology.orchestra.run.trial103.stderr: --verbose -- /var/log/ceph/ceph-client.admin.31704.log 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78950.log.gz 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31180.log 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74720.log 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31704.log.gz 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31180.log.gz 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51081.log 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94288.log 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74720.log.gz 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51081.log.gz 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86448.log 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74475.log 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94288.log.gz 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86448.log.gz 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49317.log 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74475.log.gz 2026-02-12T04:18:59.339 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30248.log 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49317.log.gz 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69869.log 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53874.log 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30248.log.gz 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69869.log.gz 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30621.log 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42844.log 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53874.log.gz 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30621.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30621.log.gz 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41261.log 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42844.log.gz 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70016.log 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41261.log.gz 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42599.log 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57023.log 2026-02-12T04:18:59.340 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70016.log.gz 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42599.log.gz 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84145.log 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57023.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57023.log.gz 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96542.log 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84145.log.gz 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93308.log 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82773.log 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96542.log.gz 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93308.log.gz 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56778.log 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92279.log 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82773.log.gz 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56778.log.gz 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45688.log 2026-02-12T04:18:59.341 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91397.log 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92279.log.gz 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37928.log 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45688.log.gz 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42158.log 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91397.log.gz 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94974.log 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37928.log.gz 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42158.log.gz 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92475.log 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73299.log 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94974.log.gz 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92475.log.gz 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80029.log 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70457.log 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr: 30.2% -- replaced with /var/log/ceph/ceph-client.admin.73299.log.gz 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80029.log.gz 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63643.log 2026-02-12T04:18:59.342 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83263.log 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr: 29.5% -- replaced with /var/log/ceph/ceph-client.admin.70457.log.gz 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63643.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63643.log.gz 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81450.log 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51669.log 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83263.log.gz 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37682.log 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81450.log.gz 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51669.log.gz 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86056.log 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77317.log 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37682.log.gz 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71878.log 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86056.log.gz 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5/var/log/ceph/ceph-client.admin.77317.log: --verbose -- /var/log/ceph/ceph-client.admin.61732.log 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77317.log.gz 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71878.log.gz 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35366.log 2026-02-12T04:18:59.343 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51865.log 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr: 29.2% -- replaced with /var/log/ceph/ceph-client.admin.61732.log.gz 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84929.log 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35366.log.gz 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51865.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.51865.log.gz 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61291.log 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68005.log 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84929.log.gz 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61291.log.gz 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37535.log 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89143.log 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68005.log.gz 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37437.log 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37535.log.gz 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89143.log.gz 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35415.log 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74916.log 2026-02-12T04:18:59.344 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37437.log.gz 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50689.log 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35415.log.gz 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74916.log.gz 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36359.log 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65898.log 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50689.log.gz 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36359.log.gz 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62614.log 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96003.log 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65898.log.gz 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62614.log.gz 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69820.log 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75651.log 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96003.log.gz 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69820.log.gz 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36898.log 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46325.log 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75651.log.gz 2026-02-12T04:18:59.345 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36898.log.gz 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88849.log 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37780.log 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46325.log.gz 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:gzip/var/log/ceph/ceph-client.admin.88849.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.96836.log 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88849.log.gz 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37780.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37780.log.gz 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54217.log 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39791.log 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96836.log.gz 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54217.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54217.log.gz 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86595.log 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88604.log 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39791.log.gz 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86595.log.gz 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80421.log 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32310.log 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88604.log.gz 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80421.log.gz 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76337.log 2026-02-12T04:18:59.346 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.32310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32310.log.gz 2026-02-12T04:18:59.347 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58937.log 2026-02-12T04:18:59.347 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79539.log 2026-02-12T04:18:59.347 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76337.log.gz 2026-02-12T04:18:59.347 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58937.log.gz 2026-02-12T04:18:59.347 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37388.log 2026-02-12T04:18:59.375 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93357.log 2026-02-12T04:18:59.375 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79539.log.gz 2026-02-12T04:18:59.375 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37388.log.gz 2026-02-12T04:18:59.375 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57464.log 2026-02-12T04:18:59.375 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26224.log 2026-02-12T04:18:59.375 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93357.log.gz 2026-02-12T04:18:59.375 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57464.log.gz 2026-02-12T04:18:59.375 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88555.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.26224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34951.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26224.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88555.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88555.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44168.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.34951.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.34951.log.gz /var/log/ceph/ceph-client.admin.67221.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr: 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44168.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77170.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46227.log 0.0% 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67221.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77170.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67466.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46227.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73740.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67466.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45492.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60262.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73740.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45492.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42501.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60262.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60262.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94631.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42501.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52600.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40967.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94631.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52600.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58790.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40967.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36212.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61487.log 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58790.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36212.log.gz 2026-02-12T04:18:59.376 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47599.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71976.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61487.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47599.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48383.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41898.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71976.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48383.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48383.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50983.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41898.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52551.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39252.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50983.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52551.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56471.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39252.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72417.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56471.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80323.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72417.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73152.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80323.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87722.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73152.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73152.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65506.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68299.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87722.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65506.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.65506.log.gz -- 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr: /var/log/ceph/ceph-client.admin.36604.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68299.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65261.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36604.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36604.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88751.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65261.log.gz 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63594.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41016.log 2026-02-12T04:18:59.377 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88751.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63594.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56729.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43531.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41016.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56729.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85370.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54805.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43531.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85370.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84684.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70310.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54805.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84684.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77219.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63692.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70310.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77219.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57808.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63692.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63692.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90025.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67662.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57808.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90025.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90025.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81254.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67662.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59821.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67662.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59624.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81254.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84047.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59821.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59624.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59624.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55491.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84047.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.84047.log.gz /var/log/ceph/ceph-client.admin.80568.log 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr: 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55491.log.gz 2026-02-12T04:18:59.378 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82969.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80568.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71829.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47991.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82969.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71829.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81646.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70898.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47991.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71535.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81646.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52404.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70898.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40477.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71535.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48236.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52404.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40477.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93896.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89535.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48236.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93896.log.gzgzip 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60066.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36457.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89535.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60066.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68887.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73642.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36457.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68887.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50248.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73642.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30935.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58200.log 0.0% 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50248.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30935.log.gz 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80470.log 2026-02-12T04:18:59.379 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58200.log.gzgzip 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45835.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72466.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80470.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51473.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45835.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72466.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82430.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35918.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51473.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82430.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82087.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71388.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35918.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50885.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82087.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63986.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71388.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34337.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50885.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57415.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63986.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.34337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39889.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34337.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95219.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57415.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39889.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39546.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49856.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95219.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:gzip/var/log/ceph/ceph-client.admin.39546.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34612.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39546.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42207.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49856.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.34612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42403.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34612.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30298.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42207.log.gz 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85223.log 2026-02-12T04:18:59.380 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42403.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52502.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30298.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35967.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 30.6% -- replaced with /var/log/ceph/ceph-client.admin.85223.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65604.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52502.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69673.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35967.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72515.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65604.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41310.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69673.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81940.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72515.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41310.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56618.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31702.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81940.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38124.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56618.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:gzip/var/log/ceph/ceph-client.admin.31702.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38418.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31702.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83508.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38124.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78852.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38418.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55050.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83508.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67417.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78852.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48089.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55050.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64869.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 29.5% -- replaced with /var/log/ceph/ceph-client.admin.67417.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84635.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48089.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31506.log 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64869.log.gz 2026-02-12T04:18:59.381 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50101.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84635.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41604.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31506.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88506.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50101.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88947.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41604.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32359.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88506.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44266.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88947.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.32359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38908.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32359.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44266.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49905.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75112.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38908.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47452.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49905.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83802.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75112.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80176.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47452.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94729.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83802.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42697.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80176.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57219.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94729.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53678.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42697.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49072.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57219.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70065.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53678.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50199.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49072.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77611.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70065.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93602.log 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50199.log.gz 2026-02-12T04:18:59.382 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47942.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77611.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69083.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93602.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32994.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47942.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51963.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69083.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.32994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52894.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32994.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45639.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51963.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77954.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52894.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45639.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45639.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70751.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77954.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73936.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70751.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70751.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81352.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78901.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73936.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31330.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81352.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33092.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78901.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31330.log.gzgzip 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.91348.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.33092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69918.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33092.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45001.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91348.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69918.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95317.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59968.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45001.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65800.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95317.log.gz 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68103.log 2026-02-12T04:18:59.383 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59968.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60213.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65800.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30765.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68103.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71780.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60213.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92426.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30765.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71290.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71780.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41653.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92426.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36702.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71290.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74181.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41653.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84194.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36702.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42045.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74181.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68789.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84194.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78003.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42045.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93994.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68789.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42094.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78003.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78457.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93994.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74083.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42094.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93259.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78457.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30150.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74083.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39595.log 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93259.log.gz 2026-02-12T04:18:59.384 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48187.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30150.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93455.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48187.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82381.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72564.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93455.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34952.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82381.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33159.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72564.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.34952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39105.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34952.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.33159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40673.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79490.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 95.8% -- replaced with /var/log/ceph/ceph-client.admin.33159.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39105.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51375.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40673.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95464.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79490.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72123.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51375.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55148.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95464.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51179.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72123.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49170.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55148.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54707.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51179.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60360.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49170.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54707.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70947.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67368.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60360.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91838.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70947.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30078.log 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67368.log.gz 2026-02-12T04:18:59.385 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62075.log 2026-02-12T04:18:59.386 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91838.log.gz 2026-02-12T04:18:59.386 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90123.log 2026-02-12T04:18:59.386 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30078.log.gz 2026-02-12T04:18:59.386 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75896.log 2026-02-12T04:18:59.386 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62075.log.gz 2026-02-12T04:18:59.386 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41996.log 2026-02-12T04:18:59.386 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90123.log.gz 2026-02-12T04:18:59.386 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95023.log 2026-02-12T04:18:59.386 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75896.log.gz 2026-02-12T04:18:59.386 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83998.log 2026-02-12T04:18:59.386 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41996.log.gz 2026-02-12T04:18:59.386 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33115.log 2026-02-12T04:18:59.386 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95023.log.gz 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80519.log 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83998.log.gz 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.33115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81793.log 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36163.log 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80519.log.gz 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr: 96.5% -- replaced with /var/log/ceph/ceph-client.admin.33115.log.gz 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69230.log 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81793.log.gz 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38320.log 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36163.log.gz 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96934.log 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69230.log.gz 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85027.log 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38320.log.gz 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37192.log 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96934.log.gz 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61683.log 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85027.log.gz 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89731.log 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37192.log.gz 2026-02-12T04:18:59.396 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83067.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61683.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54021.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89731.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57121.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83067.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50297.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54021.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87820.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57121.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86350.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50297.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74769.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87820.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35758.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86350.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31703.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74769.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88016.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35758.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31703.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.19642.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31703.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43874.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88016.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph.tmp-client.admin.19642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70604.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.19642.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79882.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43874.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59035.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70604.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46864.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 29.5% -- replaced with /var/log/ceph/ceph-client.admin.79882.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59035.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81058.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81891.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46864.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35562.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81058.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61830.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81891.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34809.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35562.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70800.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61830.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.34809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87673.log 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34809.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70800.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.47109.log -- replaced with /var/log/ceph/ceph-client.admin.70800.log.gz 2026-02-12T04:18:59.397 INFO:teuthology.orchestra.run.trial103.stderr: 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34808.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87673.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66192.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47109.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.34808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69967.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34808.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46080.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66192.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35513.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69967.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40869.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46080.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45590.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35513.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95366.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40869.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45100.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45590.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51571.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95366.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80862.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45100.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69426.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51571.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92181.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80862.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69426.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77856.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82920.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92181.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75063.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77856.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92230.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82920.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85811.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75063.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96248.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92230.log: /var/log/ceph/ceph-client.admin.85811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92230.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31280.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85811.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62124.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96248.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31280.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76680.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65555.log 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62124.log.gz 2026-02-12T04:18:59.398 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81842.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76680.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55393.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65555.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55001.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81842.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68691.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55393.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91495.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55001.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80225.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68691.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96444.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91495.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63349.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80225.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73495.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96444.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45737.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63349.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47403.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73495.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57513.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45737.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71437.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47403.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49709.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57513.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53580.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71437.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91054.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49709.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74328.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53580.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79343.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91054.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39693.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74328.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81989.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79343.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38663.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39693.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92916.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81989.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93014.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38663.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45345.log 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92916.log.gz 2026-02-12T04:18:59.399 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31033.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93014.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92818.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45345.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65751.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31033.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87281.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92818.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48677.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65751.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31509.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87281.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92622.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48677.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90270.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31509.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92622.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61977.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41114.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90270.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47844.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61977.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41114.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63104.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87771.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47844.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37731.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63104.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75210.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87771.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37339.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37731.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30984.log 0.0% 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.75210.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64820.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37339.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65359.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30984.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64820.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86889.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73250.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65359.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94337.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86889.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52747.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73250.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57611.log 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94337.log.gz 2026-02-12T04:18:59.400 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41751.log 2026-02-12T04:18:59.401 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52747.log.gz 2026-02-12T04:18:59.401 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74867.log 2026-02-12T04:18:59.401 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57611.log.gz 2026-02-12T04:18:59.401 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45982.log 2026-02-12T04:18:59.401 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41751.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58300.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74867.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75798.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45982.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81499.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 27.2% -- replaced with /var/log/ceph/ceph-client.admin.58300.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35008.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75798.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90368.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81499.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83018.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35008.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71339.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90368.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85860.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83018.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86007.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71339.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80960.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85860.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35317.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86007.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68397.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80960.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84782.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35317.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93161.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68397.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83949.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84782.log.gz 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45443.log 2026-02-12T04:18:59.405 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93161.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43089.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83949.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84292.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45443.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49758.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43089.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91985.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84292.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49268.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49758.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78703.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91985.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71927.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49268.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60899.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78703.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67907.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71927.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87232.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60899.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53629.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67907.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87232.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.67613.log -- replaced with /var/log/ceph/ceph-client.admin.87232.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83606.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53629.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67613.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44658.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94190.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83606.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48530.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44658.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48579.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94190.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49219.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48530.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86742.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48579.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54511.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49219.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78408.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 30.3% -- replaced with /var/log/ceph/ceph-client.admin.86742.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33280.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54511.log.gz 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30199.log 2026-02-12T04:18:59.406 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78408.log.gz 2026-02-12T04:18:59.407 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.33280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82626.log 2026-02-12T04:18:59.407 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33280.log.gz 2026-02-12T04:18:59.407 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54315.log 2026-02-12T04:18:59.407 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30199.log.gz 2026-02-12T04:18:59.407 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84831.log 2026-02-12T04:18:59.407 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82626.log.gz 2026-02-12T04:18:59.407 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74377.log 2026-02-12T04:18:59.407 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54315.log.gz 2026-02-12T04:18:59.407 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42795.log 2026-02-12T04:18:59.407 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84831.log.gz 2026-02-12T04:18:59.407 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73054.log 2026-02-12T04:18:59.407 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74377.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58496.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42795.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90515.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73054.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89290.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58496.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58496.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55981.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90515.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89290.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32847.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65653.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55981.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.32847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47158.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32847.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47011.log 0.0% 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65653.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69624.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47158.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38810.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47011.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56827.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69624.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75602.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38810.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56827.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.56827.log.gz -- 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr: /var/log/ceph/ceph-client.admin.96983.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75602.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95562.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96983.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74132.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95562.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88702.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31379.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74132.log.gz 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46374.log 2026-02-12T04:18:59.409 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88702.log.gz 2026-02-12T04:18:59.410 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41800.log 2026-02-12T04:18:59.410 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31379.log.gz 2026-02-12T04:18:59.410 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51816.log 2026-02-12T04:18:59.410 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46374.log.gz 2026-02-12T04:18:59.410 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62663.log 2026-02-12T04:18:59.410 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41800.log.gz 2026-02-12T04:18:59.410 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89829.log 2026-02-12T04:18:59.410 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51816.log.gz 2026-02-12T04:18:59.410 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91887.log 2026-02-12T04:18:59.410 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62663.log.gz 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92083.log 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89829.log.gz 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35208.log 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91887.log.gz 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49562.log 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92083.log.gz 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91544.log 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35208.log.gz 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81597.log 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49562.log.gz 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39007.log 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91544.log.gz 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60017.log 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81597.log.gz 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55442.log 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39007.log.gz 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64378.log 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60017.log.gz 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79931.log 2026-02-12T04:18:59.411 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55442.log.gz 2026-02-12T04:18:59.412 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30426.log 2026-02-12T04:18:59.412 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64378.log.gz 2026-02-12T04:18:59.412 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56876.log 2026-02-12T04:18:59.412 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79931.log.gz 2026-02-12T04:18:59.412 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30426.log.gz 2026-02-12T04:18:59.412 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88114.log 2026-02-12T04:18:59.412 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90662.log 2026-02-12T04:18:59.412 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56876.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56876.log.gz 2026-02-12T04:18:59.412 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31508.log 2026-02-12T04:18:59.412 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88114.log.gz 2026-02-12T04:18:59.412 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54119.log 2026-02-12T04:18:59.412 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90662.log.gz 2026-02-12T04:18:59.413 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77758.log 2026-02-12T04:18:59.413 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31508.log.gz 2026-02-12T04:18:59.413 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56079.log 2026-02-12T04:18:59.413 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54119.log.gz 2026-02-12T04:18:59.413 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75308.log 2026-02-12T04:18:59.413 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77758.log.gz 2026-02-12T04:18:59.413 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84488.log 2026-02-12T04:18:59.413 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56079.log.gz 2026-02-12T04:18:59.413 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61928.log 2026-02-12T04:18:59.413 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75308.log.gz 2026-02-12T04:18:59.414 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79637.log 2026-02-12T04:18:59.414 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84488.log.gz 2026-02-12T04:18:59.414 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36947.log 2026-02-12T04:18:59.414 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61928.log.gz 2026-02-12T04:18:59.414 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57072.log 2026-02-12T04:18:59.414 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79637.log.gz 2026-02-12T04:18:59.414 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86791.log 2026-02-12T04:18:59.414 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36947.log.gz 2026-02-12T04:18:59.414 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26077.log 2026-02-12T04:18:59.415 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57072.log.gz 2026-02-12T04:18:59.415 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94582.log 2026-02-12T04:18:59.415 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86791.log.gz 2026-02-12T04:18:59.415 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.26077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37633.log 2026-02-12T04:18:59.415 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26077.log.gz 2026-02-12T04:18:59.415 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50934.log 2026-02-12T04:18:59.415 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94582.log.gz 2026-02-12T04:18:59.415 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71633.log 2026-02-12T04:18:59.415 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37633.log.gz 2026-02-12T04:18:59.415 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91691.log 2026-02-12T04:18:59.415 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50934.log.gz 2026-02-12T04:18:59.415 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62222.log 2026-02-12T04:18:59.415 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71633.log.gz 2026-02-12T04:18:59.416 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53090.log 2026-02-12T04:18:59.416 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91691.log.gz 2026-02-12T04:18:59.416 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54560.log 2026-02-12T04:18:59.416 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62222.log.gz 2026-02-12T04:18:59.416 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77513.log 2026-02-12T04:18:59.416 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53090.log.gz 2026-02-12T04:18:59.416 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94239.log 2026-02-12T04:18:59.416 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54560.log.gz 2026-02-12T04:18:59.416 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77513.log.gz 2026-02-12T04:18:59.416 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73397.log 2026-02-12T04:18:59.417 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43776.log 2026-02-12T04:18:59.417 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94239.log.gz 2026-02-12T04:18:59.417 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86301.log 2026-02-12T04:18:59.417 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73397.log.gz 2026-02-12T04:18:59.417 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84733.log 2026-02-12T04:18:59.417 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43776.log.gz 2026-02-12T04:18:59.417 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46521.log 2026-02-12T04:18:59.417 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86301.log.gz 2026-02-12T04:18:59.417 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89878.log 2026-02-12T04:18:59.417 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84733.log.gz 2026-02-12T04:18:59.418 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39448.log 2026-02-12T04:18:59.418 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46521.log.gz 2026-02-12T04:18:59.418 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43678.log 2026-02-12T04:18:59.418 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89878.log.gz 2026-02-12T04:18:59.418 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76484.log 2026-02-12T04:18:59.418 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39448.log.gz 2026-02-12T04:18:59.418 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85713.log 2026-02-12T04:18:59.418 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43678.log.gz 2026-02-12T04:18:59.418 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55540.log 2026-02-12T04:18:59.419 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76484.log.gz 2026-02-12T04:18:59.419 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48628.log 2026-02-12T04:18:59.419 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85713.log.gz 2026-02-12T04:18:59.419 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55540.log.gz 2026-02-12T04:18:59.419 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63741.log 2026-02-12T04:18:59.419 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92573.log 2026-02-12T04:18:59.419 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48628.log.gz 2026-02-12T04:18:59.419 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91740.log 2026-02-12T04:18:59.419 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63741.log.gz 2026-02-12T04:18:59.420 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49415.log 2026-02-12T04:18:59.420 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92573.log.gz 2026-02-12T04:18:59.420 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84880.log 2026-02-12T04:18:59.420 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91740.log.gz 2026-02-12T04:18:59.420 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59722.log 2026-02-12T04:18:59.420 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49415.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49415.log.gz 2026-02-12T04:18:59.420 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95072.log 2026-02-12T04:18:59.420 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61781.log 2026-02-12T04:18:59.420 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84880.log.gz 2026-02-12T04:18:59.420 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59722.log.gz 2026-02-12T04:18:59.421 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96395.log 2026-02-12T04:18:59.421 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95072.log.gz 2026-02-12T04:18:59.421 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51326.log 2026-02-12T04:18:59.421 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61781.log.gz 2026-02-12T04:18:59.421 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34613.log 2026-02-12T04:18:59.421 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96395.log.gz 2026-02-12T04:18:59.421 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85958.log 2026-02-12T04:18:59.421 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51326.log.gz 2026-02-12T04:18:59.421 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.34613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72368.log 2026-02-12T04:18:59.421 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34613.log.gz 2026-02-12T04:18:59.421 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86203.log 2026-02-12T04:18:59.421 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85958.log.gz 2026-02-12T04:18:59.422 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70702.log 2026-02-12T04:18:59.422 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72368.log.gz 2026-02-12T04:18:59.422 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50493.log 2026-02-12T04:18:59.422 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86203.log.gz 2026-02-12T04:18:59.422 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42746.log 2026-02-12T04:18:59.422 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70702.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70702.log.gz 2026-02-12T04:18:59.422 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48040.log 2026-02-12T04:18:59.422 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50493.log.gz 2026-02-12T04:18:59.422 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84341.log 2026-02-12T04:18:59.422 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42746.log.gz 2026-02-12T04:18:59.423 INFO:teuthology.orchestra.run.trial103.stderr:gzip/var/log/ceph/ceph-client.admin.48040.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49366.log 2026-02-12T04:18:59.423 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48040.log.gz 2026-02-12T04:18:59.423 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75406.log 2026-02-12T04:18:59.423 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84341.log.gz 2026-02-12T04:18:59.423 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70212.log/var/log/ceph/ceph-client.admin.49366.log: 2026-02-12T04:18:59.423 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49366.log.gz 2026-02-12T04:18:59.423 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59231.log 2026-02-12T04:18:59.423 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75406.log.gz 2026-02-12T04:18:59.423 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90221.log 2026-02-12T04:18:59.423 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70212.log.gz 2026-02-12T04:18:59.424 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53923.log 2026-02-12T04:18:59.424 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59231.log.gz 2026-02-12T04:18:59.424 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56520.log 2026-02-12T04:18:59.424 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90221.log.gz 2026-02-12T04:18:59.424 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53384.log 2026-02-12T04:18:59.424 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53923.log.gz 2026-02-12T04:18:59.424 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70555.log 2026-02-12T04:18:59.424 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56520.log.gz 2026-02-12T04:18:59.424 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76190.log 2026-02-12T04:18:59.424 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53384.log.gz 2026-02-12T04:18:59.424 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82136.log 2026-02-12T04:18:59.425 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70555.log.gz 2026-02-12T04:18:59.425 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79833.log 2026-02-12T04:18:59.425 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76190.log.gz 2026-02-12T04:18:59.425 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46913.log 2026-02-12T04:18:59.425 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82136.log.gz 2026-02-12T04:18:59.425 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82675.log 2026-02-12T04:18:59.425 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79833.log.gz 2026-02-12T04:18:59.425 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61879.log 2026-02-12T04:18:59.425 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46913.log.gz 2026-02-12T04:18:59.426 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74965.log 2026-02-12T04:18:59.426 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82675.log.gz 2026-02-12T04:18:59.426 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78604.log 2026-02-12T04:18:59.426 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61879.log.gz 2026-02-12T04:18:59.426 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54658.log 2026-02-12T04:18:59.426 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74965.log.gz 2026-02-12T04:18:59.426 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36506.log 2026-02-12T04:18:59.426 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78604.log.gz 2026-02-12T04:18:59.426 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71486.log 2026-02-12T04:18:59.426 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54658.log.gz 2026-02-12T04:18:59.426 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36506.log.gz 2026-02-12T04:18:59.426 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42893.log 2026-02-12T04:18:59.427 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64329.log 2026-02-12T04:18:59.427 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71486.log.gz 2026-02-12T04:18:59.427 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42893.log.gz 2026-02-12T04:18:59.427 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40526.log 2026-02-12T04:18:59.427 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64329.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64329.log.gz 2026-02-12T04:18:59.427 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43580.log 2026-02-12T04:18:59.427 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40526.log.gz 2026-02-12T04:18:59.427 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48726.log 2026-02-12T04:18:59.428 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43580.log.gz 2026-02-12T04:18:59.428 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32945.log 2026-02-12T04:18:59.428 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48726.log.gz 2026-02-12T04:18:59.428 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87428.log 2026-02-12T04:18:59.428 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.32945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32945.log.gz 2026-02-12T04:18:59.428 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70849.log 2026-02-12T04:18:59.428 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87428.log.gz 2026-02-12T04:18:59.428 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32896.log 2026-02-12T04:18:59.429 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70849.log.gz 2026-02-12T04:18:59.429 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78654.log 2026-02-12T04:18:59.429 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.32896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32896.log.gz 2026-02-12T04:18:59.429 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60164.log 2026-02-12T04:18:59.429 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78654.log.gz 2026-02-12T04:18:59.429 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60703.log 2026-02-12T04:18:59.429 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60164.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60164.log.gz 2026-02-12T04:18:59.430 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90417.log 2026-02-12T04:18:59.430 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60703.log.gz 2026-02-12T04:18:59.430 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47305.log 2026-02-12T04:18:59.430 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90417.log.gz 2026-02-12T04:18:59.430 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65065.log 2026-02-12T04:18:59.430 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47305.log.gz 2026-02-12T04:18:59.430 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93406.log 2026-02-12T04:18:59.430 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65065.log.gz 2026-02-12T04:18:59.431 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51914.log 2026-02-12T04:18:59.431 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93406.log.gz 2026-02-12T04:18:59.431 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45786.log 2026-02-12T04:18:59.431 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51914.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51914.log.gz 2026-02-12T04:18:59.431 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86105.log 2026-02-12T04:18:59.431 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45786.log.gz 2026-02-12T04:18:59.431 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92377.log 2026-02-12T04:18:59.432 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86105.log.gz 2026-02-12T04:18:59.432 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65457.log 2026-02-12T04:18:59.432 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92377.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92377.log.gz 2026-02-12T04:18:59.432 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44119.log 2026-02-12T04:18:59.432 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65457.log.gz 2026-02-12T04:18:59.432 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71192.log 2026-02-12T04:18:59.432 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44119.log.gz 2026-02-12T04:18:59.432 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52943.log 2026-02-12T04:18:59.433 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71192.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.71192.log.gz /var/log/ceph/ceph-client.admin.96493.log 2026-02-12T04:18:59.433 INFO:teuthology.orchestra.run.trial103.stderr: 2026-02-12T04:18:59.433 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52943.log.gz 2026-02-12T04:18:59.433 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72270.log 2026-02-12T04:18:59.433 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54364.log 2026-02-12T04:18:59.433 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96493.log.gz 2026-02-12T04:18:59.434 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51522.log 2026-02-12T04:18:59.434 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72270.log.gz 2026-02-12T04:18:59.434 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54266.log 2026-02-12T04:18:59.434 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54364.log.gz 2026-02-12T04:18:59.434 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67515.log 2026-02-12T04:18:59.434 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51522.log.gz 2026-02-12T04:18:59.434 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34807.log 2026-02-12T04:18:59.434 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54266.log.gz 2026-02-12T04:18:59.434 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26318.log 2026-02-12T04:18:59.434 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67515.log.gz 2026-02-12T04:18:59.435 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56324.log 2026-02-12T04:18:59.435 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.34807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34807.log.gz 2026-02-12T04:18:59.435 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.26318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39301.log 2026-02-12T04:18:59.435 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26318.log.gz 2026-02-12T04:18:59.435 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38075.log 2026-02-12T04:18:59.435 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56324.log.gz 2026-02-12T04:18:59.435 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39301.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.39301.log.gz -- 2026-02-12T04:18:59.435 INFO:teuthology.orchestra.run.trial103.stderr: /var/log/ceph/ceph-client.admin.53286.log 2026-02-12T04:18:59.436 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46276.log 2026-02-12T04:18:59.436 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38075.log.gz 2026-02-12T04:18:59.436 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58004.log 2026-02-12T04:18:59.436 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53286.log.gz 2026-02-12T04:18:59.436 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59427.log 2026-02-12T04:18:59.436 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46276.log.gz 2026-02-12T04:18:59.436 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82871.log 2026-02-12T04:18:59.436 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58004.log.gz 2026-02-12T04:18:59.436 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83851.log 2026-02-12T04:18:59.436 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59427.log.gz 2026-02-12T04:18:59.436 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48974.log 2026-02-12T04:18:59.436 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82871.log.gz 2026-02-12T04:18:59.436 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77121.log 2026-02-12T04:18:59.436 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83851.log.gz 2026-02-12T04:18:59.437 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85321.log 2026-02-12T04:18:59.437 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48974.log.gz 2026-02-12T04:18:59.437 INFO:teuthology.orchestra.run.trial103.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-12T04:18:59.437 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77121.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.77121.log.gz /var/log/ceph/ceph-client.admin.96052.log 2026-02-12T04:18:59.437 INFO:teuthology.orchestra.run.trial103.stderr: 2026-02-12T04:18:59.437 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55638.log 2026-02-12T04:18:59.437 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85321.log.gz 2026-02-12T04:18:59.437 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64525.log 2026-02-12T04:18:59.437 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96052.log.gz 2026-02-12T04:18:59.438 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39987.log 2026-02-12T04:18:59.438 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55638.log.gz 2026-02-12T04:18:59.438 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87918.log 2026-02-12T04:18:59.438 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64525.log.gz 2026-02-12T04:18:59.438 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87869.log 2026-02-12T04:18:59.438 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39987.log.gz 2026-02-12T04:18:59.438 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41163.log 2026-02-12T04:18:59.438 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87918.log.gz 2026-02-12T04:18:59.439 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50836.log 2026-02-12T04:18:59.439 INFO:teuthology.orchestra.run.trial103.stderr: 29.2% -- replaced with /var/log/ceph/ceph-client.admin.87869.log.gz 2026-02-12T04:18:59.439 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72956.log 2026-02-12T04:18:59.439 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41163.log.gz 2026-02-12T04:18:59.439 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36555.log 2026-02-12T04:18:59.439 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50836.log.gz 2026-02-12T04:18:59.439 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95415.log 2026-02-12T04:18:59.439 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72956.log.gz 2026-02-12T04:18:59.439 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63790.log 2026-02-12T04:18:59.439 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36555.log.gz 2026-02-12T04:18:59.440 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83704.log 2026-02-12T04:18:59.440 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95415.log.gz 2026-02-12T04:18:59.440 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63790.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63790.log.gz 2026-02-12T04:18:59.440 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35820.log 2026-02-12T04:18:59.440 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86693.log 2026-02-12T04:18:59.440 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83704.log.gz 2026-02-12T04:18:59.440 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56177.log 2026-02-12T04:18:59.440 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35820.log.gz 2026-02-12T04:18:59.440 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86693.log.gz 2026-02-12T04:18:59.440 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73103.log 2026-02-12T04:18:59.440 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34610.log 2026-02-12T04:18:59.440 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56177.log.gz 2026-02-12T04:18:59.440 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73103.log.gz 2026-02-12T04:18:59.441 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38712.log 2026-02-12T04:18:59.441 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.34610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78065.log 2026-02-12T04:18:59.441 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34610.log.gz 2026-02-12T04:18:59.441 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33329.log 2026-02-12T04:18:59.441 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38712.log.gz 2026-02-12T04:18:59.441 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50542.log 2026-02-12T04:18:59.441 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78065.log.gz 2026-02-12T04:18:59.441 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.33329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91299.log 2026-02-12T04:18:59.441 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33329.log.gz 2026-02-12T04:18:59.441 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86546.log 2026-02-12T04:18:59.442 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50542.log.gz 2026-02-12T04:18:59.442 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85468.log 2026-02-12T04:18:59.442 INFO:teuthology.orchestra.run.trial103.stderr: 28.6% -- replaced with /var/log/ceph/ceph-client.admin.91299.log.gz 2026-02-12T04:18:59.442 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86546.log.gz 2026-02-12T04:18:59.442 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53335.log 2026-02-12T04:18:59.442 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87575.log 2026-02-12T04:18:59.442 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85468.log.gz 2026-02-12T04:18:59.442 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85664.log 2026-02-12T04:18:59.442 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53335.log.gz 2026-02-12T04:18:59.442 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87575.log.gz 2026-02-12T04:18:59.442 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49660.log 2026-02-12T04:18:59.443 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85664.log.gz 2026-02-12T04:18:59.443 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55197.log 2026-02-12T04:18:59.443 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61193.log 2026-02-12T04:18:59.443 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49660.log.gz 2026-02-12T04:18:59.443 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55197.log.gz 2026-02-12T04:18:59.443 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91152.log 2026-02-12T04:18:59.443 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61193.log.gz 2026-02-12T04:18:59.443 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74622.log 2026-02-12T04:18:59.444 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91152.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91152.log.gz 2026-02-12T04:18:59.444 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52012.log 2026-02-12T04:18:59.444 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74622.log.gz 2026-02-12T04:18:59.444 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50787.log 2026-02-12T04:18:59.444 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52012.log.gz 2026-02-12T04:18:59.444 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32261.log 2026-02-12T04:18:59.444 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50787.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.39938.log 2026-02-12T04:18:59.444 INFO:teuthology.orchestra.run.trial103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50787.log.gz 2026-02-12T04:18:59.445 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.32261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32261.log.gz 2026-02-12T04:18:59.445 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36016.log 2026-02-12T04:18:59.445 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39938.log.gz 2026-02-12T04:18:59.445 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52355.log 2026-02-12T04:18:59.445 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36016.log.gz 2026-02-12T04:18:59.445 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50346.log 2026-02-12T04:18:59.445 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52355.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52355.log.gz 2026-02-12T04:18:59.445 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58741.log 2026-02-12T04:18:59.446 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50346.log.gz 2026-02-12T04:18:59.446 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81695.log 2026-02-12T04:18:59.446 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83214.log 2026-02-12T04:18:59.446 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58741.log.gz 2026-02-12T04:18:59.446 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81695.log.gz 2026-02-12T04:18:59.446 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65947.log 2026-02-12T04:18:59.446 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83214.log.gz 2026-02-12T04:18:59.447 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54070.log 2026-02-12T04:18:59.447 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65947.log.gz 2026-02-12T04:18:59.447 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76288.log 2026-02-12T04:18:59.447 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83116.log 0.0% 2026-02-12T04:18:59.447 INFO:teuthology.orchestra.run.trial103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54070.log.gz 2026-02-12T04:18:59.447 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76288.log.gz 2026-02-12T04:18:59.447 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82234.log 2026-02-12T04:18:59.448 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64133.log 2026-02-12T04:18:59.448 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83116.log.gz 2026-02-12T04:18:59.448 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82234.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82234.log.gz 2026-02-12T04:18:59.448 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66584.log 2026-02-12T04:18:59.448 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31131.log 2026-02-12T04:18:59.448 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64133.log.gz 2026-02-12T04:18:59.448 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66584.log.gz 2026-02-12T04:18:59.448 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31705.log 2026-02-12T04:18:59.449 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31131.log.gz 2026-02-12T04:18:59.449 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96640.log 2026-02-12T04:18:59.449 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31705.log.gz 2026-02-12T04:18:59.449 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48481.log 2026-02-12T04:18:59.449 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96640.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96640.log.gz 2026-02-12T04:18:59.449 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32798.log 2026-02-12T04:18:59.449 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48481.log.gz 2026-02-12T04:18:59.449 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90564.log 2026-02-12T04:18:59.450 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.32798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32798.log.gz 2026-02-12T04:18:59.450 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31932.log 2026-02-12T04:18:59.450 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90564.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90564.log.gz 2026-02-12T04:18:59.450 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94827.log 2026-02-12T04:18:59.450 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31932.log.gz 2026-02-12T04:18:59.450 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31082.log 2026-02-12T04:18:59.450 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94827.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94827.log.gz 2026-02-12T04:18:59.450 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59870.log 2026-02-12T04:18:59.451 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31082.log.gz 2026-02-12T04:18:59.451 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89584.log 2026-02-12T04:18:59.451 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59870.log.gz 2026-02-12T04:18:59.451 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66486.log 2026-02-12T04:18:59.451 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89584.log.gz 2026-02-12T04:18:59.451 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37486.log 2026-02-12T04:18:59.451 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78752.log 2026-02-12T04:18:59.451 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66486.log.gz 2026-02-12T04:18:59.452 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37486.log.gz 2026-02-12T04:18:59.452 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53482.log 2026-02-12T04:18:59.452 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60948.log 2026-02-12T04:18:59.452 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78752.log.gz 2026-02-12T04:18:59.452 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53482.log.gz 2026-02-12T04:18:59.452 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36751.log 2026-02-12T04:18:59.452 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26518.log 2026-02-12T04:18:59.452 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60948.log.gz 2026-02-12T04:18:59.453 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36751.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36751.log.gz 2026-02-12T04:18:59.453 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89976.log 2026-02-12T04:18:59.453 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.26518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89927.log 2026-02-12T04:18:59.453 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26518.log.gz 2026-02-12T04:18:59.453 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89976.log.gz 2026-02-12T04:18:59.453 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61340.log 2026-02-12T04:18:59.453 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89927.log.gz 2026-02-12T04:18:59.454 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44903.log 2026-02-12T04:18:59.454 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61340.log.gz 2026-02-12T04:18:59.454 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36065.log 2026-02-12T04:18:59.454 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44903.log.gz 2026-02-12T04:18:59.454 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70653.log 2026-02-12T04:18:59.454 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36065.log.gz 2026-02-12T04:18:59.454 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88996.log 2026-02-12T04:18:59.454 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70653.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.70653.log.gz --verbose 2026-02-12T04:18:59.455 INFO:teuthology.orchestra.run.trial103.stderr: -- /var/log/ceph/ceph-client.admin.51718.log 2026-02-12T04:18:59.455 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88996.log.gz 2026-02-12T04:18:59.455 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91642.log 2026-02-12T04:18:59.455 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51718.log.gz 2026-02-12T04:18:59.455 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59280.log 2026-02-12T04:18:59.455 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91642.log.gz 2026-02-12T04:18:59.455 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66535.log 2026-02-12T04:18:59.455 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59280.log.gz 2026-02-12T04:18:59.455 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40379.log 2026-02-12T04:18:59.456 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66535.log.gz 2026-02-12T04:18:59.456 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59378.log 2026-02-12T04:18:59.456 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40379.log.gz 2026-02-12T04:18:59.456 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37977.log 2026-02-12T04:18:59.456 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59378.log.gz 2026-02-12T04:18:59.456 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39154.log 2026-02-12T04:18:59.456 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37977.log.gz 2026-02-12T04:18:59.456 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36114.log 2026-02-12T04:18:59.457 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39154.log.gz 2026-02-12T04:18:59.457 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81205.log 2026-02-12T04:18:59.457 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36114.log.gz 2026-02-12T04:18:59.457 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66780.log 2026-02-12T04:18:59.457 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81205.log.gz 2026-02-12T04:18:59.457 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32099.log 2026-02-12T04:18:59.457 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66780.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66780.log.gz 2026-02-12T04:18:59.457 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41457.log 2026-02-12T04:18:59.458 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.32099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32099.log.gz 2026-02-12T04:18:59.458 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39056.log 2026-02-12T04:18:59.458 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41457.log.gz 2026-02-12T04:18:59.458 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45884.log 2026-02-12T04:18:59.458 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39056.log.gz 2026-02-12T04:18:59.458 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85076.log 2026-02-12T04:18:59.458 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45884.log.gz 2026-02-12T04:18:59.458 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80617.log 2026-02-12T04:18:59.459 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85076.log.gz 2026-02-12T04:18:59.459 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69475.log 2026-02-12T04:18:59.459 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80617.log.gz 2026-02-12T04:18:59.459 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87036.log 2026-02-12T04:18:59.459 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69475.log: 29.5% -- replaced with /var/log/ceph/ceph-client.admin.69475.log.gz 2026-02-12T04:18:59.459 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37143.log 2026-02-12T04:18:59.459 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87036.log.gz 2026-02-12T04:18:59.459 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85517.log 2026-02-12T04:18:59.460 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37143.log.gz 2026-02-12T04:18:59.460 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89780.log 2026-02-12T04:18:59.460 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85517.log.gz 2026-02-12T04:18:59.460 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91789.log 2026-02-12T04:18:59.460 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89780.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89780.log.gz 2026-02-12T04:18:59.460 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66388.log 2026-02-12T04:18:59.460 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34810.log 2026-02-12T04:18:59.460 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91789.log.gz 2026-02-12T04:18:59.461 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66388.log.gz 2026-02-12T04:18:59.461 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36310.log 2026-02-12T04:18:59.461 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60409.log 2026-02-12T04:18:59.461 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.34810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34810.log.gz 2026-02-12T04:18:59.461 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74524.log 2026-02-12T04:18:59.461 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36310.log.gz 2026-02-12T04:18:59.461 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60409.log.gz 2026-02-12T04:18:59.461 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65016.log 2026-02-12T04:18:59.461 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92769.log 2026-02-12T04:18:59.461 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74524.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74524.log.gz 2026-02-12T04:18:59.462 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96297.log 2026-02-12T04:18:59.462 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65016.log.gz 2026-02-12T04:18:59.462 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57317.log 2026-02-12T04:18:59.462 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92769.log.gz 2026-02-12T04:18:59.462 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55295.log 2026-02-12T04:18:59.462 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96297.log.gz 2026-02-12T04:18:59.462 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92867.log 2026-02-12T04:18:59.462 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57317.log.gz 2026-02-12T04:18:59.462 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66290.log 2026-02-12T04:18:59.462 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55295.log.gz 2026-02-12T04:18:59.463 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54854.log 2026-02-12T04:18:59.463 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92867.log.gz 2026-02-12T04:18:59.463 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79980.log 2026-02-12T04:18:59.463 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66290.log.gz 2026-02-12T04:18:59.463 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44707.log 2026-02-12T04:18:59.463 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54854.log.gz 2026-02-12T04:18:59.463 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88800.log 2026-02-12T04:18:59.463 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79980.log.gz 2026-02-12T04:18:59.463 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96885.log 2026-02-12T04:18:59.463 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44707.log.gz 2026-02-12T04:18:59.464 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42305.log 2026-02-12T04:18:59.464 INFO:teuthology.orchestra.run.trial103.stderr: 29.2% -- replaced with /var/log/ceph/ceph-client.admin.88800.log.gz 2026-02-12T04:18:59.464 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84096.log 2026-02-12T04:18:59.464 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96885.log.gz 2026-02-12T04:18:59.464 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64231.log 2026-02-12T04:18:59.464 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42305.log.gz 2026-02-12T04:18:59.464 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64574.log 2026-02-12T04:18:59.464 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84096.log.gz 2026-02-12T04:18:59.464 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71143.log 2026-02-12T04:18:59.464 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64231.log.gz 2026-02-12T04:18:59.465 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49954.log 2026-02-12T04:18:59.465 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64574.log.gz 2026-02-12T04:18:59.465 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71143.log.gz 2026-02-12T04:18:59.465 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32382.log 2026-02-12T04:18:59.465 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61046.log 2026-02-12T04:18:59.465 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49954.log.gz 2026-02-12T04:18:59.465 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.32382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58349.log 2026-02-12T04:18:59.465 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61046.log.gz 2026-02-12T04:18:59.465 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42452.log 2026-02-12T04:18:59.465 INFO:teuthology.orchestra.run.trial103.stderr: 96.1% -- replaced with /var/log/ceph/ceph-client.admin.32382.log.gz 2026-02-12T04:18:59.466 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58349.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58349.log.gz 2026-02-12T04:18:59.466 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39742.log 2026-02-12T04:18:59.466 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86938.log 2026-02-12T04:18:59.466 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42452.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42452.log.gz 2026-02-12T04:18:59.466 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50738.log 2026-02-12T04:18:59.466 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39742.log.gz 2026-02-12T04:18:59.466 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56226.log 2026-02-12T04:18:59.466 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86938.log.gz 2026-02-12T04:18:59.466 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70506.log 2026-02-12T04:18:59.466 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50738.log.gz 2026-02-12T04:18:59.467 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56226.log.gz 2026-02-12T04:18:59.467 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77905.log 2026-02-12T04:18:59.467 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32043.log 2026-02-12T04:18:59.467 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70506.log.gz 2026-02-12T04:18:59.467 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37584.log 2026-02-12T04:18:59.467 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77905.log.gz 2026-02-12T04:18:59.467 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.32043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94925.log 2026-02-12T04:18:59.467 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32043.log.gz 2026-02-12T04:18:59.467 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39203.log 2026-02-12T04:18:59.467 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37584.log.gz 2026-02-12T04:18:59.468 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53237.log 2026-02-12T04:18:59.468 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94925.log.gz 2026-02-12T04:18:59.468 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85566.log 2026-02-12T04:18:59.468 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39203.log.gz 2026-02-12T04:18:59.468 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58053.log 2026-02-12T04:18:59.468 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53237.log.gz 2026-02-12T04:18:59.468 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63398.log 2026-02-12T04:18:59.468 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85566.log.gz 2026-02-12T04:18:59.468 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37045.log 2026-02-12T04:18:59.469 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58053.log.gz 2026-02-12T04:18:59.469 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87183.log 2026-02-12T04:18:59.469 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63398.log.gz 2026-02-12T04:18:59.469 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66339.log 2026-02-12T04:18:59.469 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37045.log.gz 2026-02-12T04:18:59.469 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81156.log 2026-02-12T04:18:59.469 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87183.log.gz 2026-02-12T04:18:59.469 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83655.log 2026-02-12T04:18:59.469 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66339.log.gz 2026-02-12T04:18:59.469 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90907.log 2026-02-12T04:18:59.469 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81156.log.gz 2026-02-12T04:18:59.470 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73201.log 2026-02-12T04:18:59.470 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83655.log.gz 2026-02-12T04:18:59.470 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33043.log 2026-02-12T04:18:59.470 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90907.log.gz 2026-02-12T04:18:59.470 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91201.log 2026-02-12T04:18:59.470 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73201.log.gz 2026-02-12T04:18:59.470 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.33043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68593.log 2026-02-12T04:18:59.470 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33043.log.gz 2026-02-12T04:18:59.470 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91201.log.gz 2026-02-12T04:18:59.470 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50003.log 2026-02-12T04:18:59.470 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63839.log 2026-02-12T04:18:59.471 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68593.log: /var/log/ceph/ceph-client.admin.50003.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.68593.log.gz 2026-02-12T04:18:59.471 INFO:teuthology.orchestra.run.trial103.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56569.log 2026-02-12T04:18:59.471 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50003.log.gz 2026-02-12T04:18:59.471 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37094.log 2026-02-12T04:18:59.471 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63839.log.gz 2026-02-12T04:18:59.471 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66927.log 2026-02-12T04:18:59.471 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56569.log.gz 2026-02-12T04:18:59.471 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37094.log.gz 2026-02-12T04:18:59.471 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83900.log 2026-02-12T04:18:59.472 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30766.log 2026-02-12T04:18:59.472 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66927.log.gz 2026-02-12T04:18:59.472 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46178.log 2026-02-12T04:18:59.472 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83900.log.gz 2026-02-12T04:18:59.472 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30766.log.gz 2026-02-12T04:18:59.472 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49464.log 2026-02-12T04:18:59.472 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74818.log 2026-02-12T04:18:59.472 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46178.log.gz 2026-02-12T04:18:59.472 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62369.log 2026-02-12T04:18:59.473 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49464.log.gz 2026-02-12T04:18:59.473 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96689.log 2026-02-12T04:18:59.473 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74818.log.gz 2026-02-12T04:18:59.473 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92965.log 2026-02-12T04:18:59.473 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62369.log.gz 2026-02-12T04:18:59.473 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56422.log 2026-02-12T04:18:59.473 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96689.log.gz 2026-02-12T04:18:59.473 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69132.log 2026-02-12T04:18:59.473 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92965.log.gz 2026-02-12T04:18:59.473 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65408.log 2026-02-12T04:18:59.473 INFO:teuthology.orchestra.run.trial103.stderr: 29.3% -- replaced with /var/log/ceph/ceph-client.admin.56422.log.gz 2026-02-12T04:18:59.474 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88065.log 2026-02-12T04:18:59.474 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69132.log.gz 2026-02-12T04:18:59.474 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40281.log 2026-02-12T04:18:59.474 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65408.log.gz 2026-02-12T04:18:59.474 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45050.log 2026-02-12T04:18:59.474 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88065.log.gz 2026-02-12T04:18:59.474 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60801.log 2026-02-12T04:18:59.474 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40281.log.gz 2026-02-12T04:18:59.474 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95807.log 2026-02-12T04:18:59.475 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45050.log.gz 2026-02-12T04:18:59.475 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60801.log.gz 2026-02-12T04:18:59.475 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66143.log 2026-02-12T04:18:59.475 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93798.log 2026-02-12T04:18:59.475 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95807.log.gz 2026-02-12T04:18:59.475 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25819.log 2026-02-12T04:18:59.475 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66143.log.gz 2026-02-12T04:18:59.475 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82332.log 2026-02-12T04:18:59.475 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93798.log.gz 2026-02-12T04:18:59.475 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.25819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65702.log 2026-02-12T04:18:59.475 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25819.log.gz 2026-02-12T04:18:59.476 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59084.log 2026-02-12T04:18:59.476 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82332.log.gz 2026-02-12T04:18:59.476 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88310.log 2026-02-12T04:18:59.476 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65702.log.gz 2026-02-12T04:18:59.476 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63300.log 2026-02-12T04:18:59.476 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59084.log.gz 2026-02-12T04:18:59.476 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95905.log 2026-02-12T04:18:59.476 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88310.log.gz 2026-02-12T04:18:59.476 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38173.log 2026-02-12T04:18:59.476 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63300.log.gz 2026-02-12T04:18:59.476 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58692.log 2026-02-12T04:18:59.476 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95905.log.gz 2026-02-12T04:18:59.477 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60605.log 2026-02-12T04:18:59.477 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38173.log.gz 2026-02-12T04:18:59.477 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58692.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-12T04:18:59.477 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58692.log.gz 2026-02-12T04:18:59.477 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58151.log 2026-02-12T04:18:59.477 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60605.log.gz 2026-02-12T04:18:59.477 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53825.log 2026-02-12T04:18:59.478 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72221.log 2026-02-12T04:18:59.478 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58151.log.gz 2026-02-12T04:18:59.478 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68250.log 2026-02-12T04:18:59.478 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53825.log.gz 2026-02-12T04:18:59.478 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65163.log 2026-02-12T04:18:59.478 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72221.log.gz 2026-02-12T04:18:59.478 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61095.log 2026-02-12T04:18:59.478 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68250.log.gz 2026-02-12T04:18:59.478 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39840.log 2026-02-12T04:18:59.478 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65163.log.gz 2026-02-12T04:18:59.478 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61095.log.gz 2026-02-12T04:18:59.478 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89045.log 2026-02-12T04:18:59.479 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54413.log 2026-02-12T04:18:59.479 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39840.log.gz 2026-02-12T04:18:59.479 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43187.log 2026-02-12T04:18:59.479 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89045.log.gz 2026-02-12T04:18:59.479 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43334.log 2026-02-12T04:18:59.479 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54413.log.gz 2026-02-12T04:18:59.479 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42256.log 2026-02-12T04:18:59.479 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43187.log.gz 2026-02-12T04:18:59.479 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93063.log 2026-02-12T04:18:59.479 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43334.log.gz 2026-02-12T04:18:59.480 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70408.log 2026-02-12T04:18:59.480 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42256.log.gz 2026-02-12T04:18:59.480 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62516.log 2026-02-12T04:18:59.480 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93063.log.gz 2026-02-12T04:18:59.480 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.70408.log: -- /var/log/ceph/ceph-client.admin.60654.log 2026-02-12T04:18:59.480 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70408.log.gz 2026-02-12T04:18:59.480 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55687.log 2026-02-12T04:18:59.480 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62516.log.gz 2026-02-12T04:18:59.480 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67074.log 2026-02-12T04:18:59.480 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60654.log.gz 2026-02-12T04:18:59.481 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89192.log 2026-02-12T04:18:59.481 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55687.log.gz 2026-02-12T04:18:59.481 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45198.log 2026-02-12T04:18:59.481 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67074.log.gz 2026-02-12T04:18:59.481 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63447.log 2026-02-12T04:18:59.481 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89192.log.gz 2026-02-12T04:18:59.481 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43138.log 2026-02-12T04:18:59.481 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45198.log.gz 2026-02-12T04:18:59.481 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63447.log.gz 2026-02-12T04:18:59.481 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46766.log 2026-02-12T04:18:59.482 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57268.log 2026-02-12T04:18:59.482 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43138.log.gz 2026-02-12T04:18:59.482 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86644.log 2026-02-12T04:18:59.482 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46766.log.gz 2026-02-12T04:18:59.482 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57268.log.gz 2026-02-12T04:18:59.482 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55589.log 2026-02-12T04:18:59.482 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35660.log 2026-02-12T04:18:59.482 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86644.log.gz 2026-02-12T04:18:59.482 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31229.log 2026-02-12T04:18:59.482 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55589.log.gz 2026-02-12T04:18:59.483 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44021.log 2026-02-12T04:18:59.483 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35660.log.gz 2026-02-12T04:18:59.483 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45394.log 2026-02-12T04:18:59.483 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31229.log.gz 2026-02-12T04:18:59.483 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61144.log 2026-02-12T04:18:59.483 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44021.log.gz 2026-02-12T04:18:59.483 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45394.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2026-02-12T04:18:59.483 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45394.log.gz 2026-02-12T04:18:59.483 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62859.log 2026-02-12T04:18:59.483 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61144.log.gz 2026-02-12T04:18:59.484 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82528.log 2026-02-12T04:18:59.484 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58545.log 2026-02-12T04:18:59.484 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62859.log.gz 2026-02-12T04:18:59.484 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82528.log.gz 2026-02-12T04:18:59.484 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79146.log 2026-02-12T04:18:59.484 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45247.log 2026-02-12T04:18:59.484 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58545.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58545.log.gz 2026-02-12T04:18:59.485 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71094.log 2026-02-12T04:18:59.485 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79146.log.gz 2026-02-12T04:18:59.485 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57710.log 2026-02-12T04:18:59.485 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45247.log.gz 2026-02-12T04:18:59.485 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71094.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.92524.log -- replaced with /var/log/ceph/ceph-client.admin.71094.log.gz 2026-02-12T04:18:59.485 INFO:teuthology.orchestra.run.trial103.stderr: 2026-02-12T04:18:59.485 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58594.log 2026-02-12T04:18:59.485 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57710.log.gz 2026-02-12T04:18:59.485 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92524.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92524.log.gz 2026-02-12T04:18:59.485 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52453.log 2026-02-12T04:18:59.486 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76778.log 2026-02-12T04:18:59.486 INFO:teuthology.orchestra.run.trial103.stderr: 95.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-12T04:18:59.486 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58594.log.gz 2026-02-12T04:18:59.486 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61438.log 2026-02-12T04:18:59.486 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52453.log.gz 2026-02-12T04:18:59.486 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76778.log.gz 2026-02-12T04:18:59.486 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73838.log 2026-02-12T04:18:59.486 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95954.log 2026-02-12T04:18:59.486 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61438.log.gz 2026-02-12T04:18:59.487 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67760.log 2026-02-12T04:18:59.487 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73838.log.gz 2026-02-12T04:18:59.487 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95954.log.gz 2026-02-12T04:18:59.487 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95513.log 2026-02-12T04:18:59.487 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65114.log 2026-02-12T04:18:59.487 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67760.log.gz 2026-02-12T04:18:59.487 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90074.log 2026-02-12T04:18:59.487 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95513.log.gz 2026-02-12T04:18:59.487 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58102.log 2026-02-12T04:18:59.487 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65114.log.gz 2026-02-12T04:18:59.488 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56030.log 2026-02-12T04:18:59.488 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90074.log.gz 2026-02-12T04:18:59.488 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58102.log.gz 2026-02-12T04:18:59.488 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81744.log 2026-02-12T04:18:59.488 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37241.log 2026-02-12T04:18:59.488 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56030.log.gz 2026-02-12T04:18:59.488 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81744.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81744.log.gz 2026-02-12T04:18:59.488 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71045.log 2026-02-12T04:18:59.489 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52208.log 2026-02-12T04:18:59.489 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37241.log.gz 2026-02-12T04:18:59.489 INFO:teuthology.orchestra.run.trial103.stderr:gzip/var/log/ceph/ceph-client.admin.71045.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.77415.log 2026-02-12T04:18:59.489 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71045.log.gz 2026-02-12T04:18:59.489 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63888.log 2026-02-12T04:18:59.489 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52208.log.gz 2026-02-12T04:18:59.489 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43481.log 2026-02-12T04:18:59.489 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77415.log.gz 2026-02-12T04:18:59.489 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89682.log 2026-02-12T04:18:59.489 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63888.log.gz 2026-02-12T04:18:59.490 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43481.log.gz 2026-02-12T04:18:59.490 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68152.log 2026-02-12T04:18:59.490 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73985.log 2026-02-12T04:18:59.490 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89682.log.gz 2026-02-12T04:18:59.490 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58986.log 2026-02-12T04:18:59.490 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68152.log.gz 2026-02-12T04:18:59.490 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73985.log.gz 2026-02-12T04:18:59.490 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71241.log 2026-02-12T04:18:59.491 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90613.log 2026-02-12T04:18:59.491 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58986.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58986.log.gz 2026-02-12T04:18:59.491 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57170.log 2026-02-12T04:18:59.491 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71241.log.gz 2026-02-12T04:18:59.491 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43432.log 2026-02-12T04:18:59.491 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90613.log.gz 2026-02-12T04:18:59.491 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26271.log 2026-02-12T04:18:59.491 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57170.log.gz 2026-02-12T04:18:59.492 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37879.log 2026-02-12T04:18:59.492 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43432.log.gz 2026-02-12T04:18:59.492 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.26271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68985.log 2026-02-12T04:18:59.492 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26271.log.gz 2026-02-12T04:18:59.492 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69328.log 2026-02-12T04:18:59.492 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37879.log.gz 2026-02-12T04:18:59.492 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59919.log 2026-02-12T04:18:59.492 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68985.log.gz 2026-02-12T04:18:59.492 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47060.log 2026-02-12T04:18:59.492 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69328.log.gz 2026-02-12T04:18:59.492 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69771.log 2026-02-12T04:18:59.492 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59919.log.gz 2026-02-12T04:18:59.493 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40085.log 2026-02-12T04:18:59.493 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47060.log.gz 2026-02-12T04:18:59.493 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45149.log 2026-02-12T04:18:59.493 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69771.log.gz 2026-02-12T04:18:59.493 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43236.log 2026-02-12T04:18:59.493 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40085.log.gz 2026-02-12T04:18:59.493 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67564.log 2026-02-12T04:18:59.493 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45149.log.gz 2026-02-12T04:18:59.493 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93553.log 2026-02-12T04:18:59.493 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43236.log.gz 2026-02-12T04:18:59.493 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67564.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67564.log.gz 2026-02-12T04:18:59.494 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83459.log 2026-02-12T04:18:59.494 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54756.log 2026-02-12T04:18:59.494 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93553.log.gz 2026-02-12T04:18:59.494 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68495.log 2026-02-12T04:18:59.494 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83459.log.gz 2026-02-12T04:18:59.494 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39644.log 2026-02-12T04:18:59.494 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54756.log.gz 2026-02-12T04:18:59.494 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85909.log 2026-02-12T04:18:59.494 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68495.log.gz 2026-02-12T04:18:59.495 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39644.log.gz 2026-02-12T04:18:59.495 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49121.log 2026-02-12T04:18:59.495 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89241.log 2026-02-12T04:18:59.495 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85909.log.gz 2026-02-12T04:18:59.495 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69377.log 2026-02-12T04:18:59.495 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49121.log.gz 2026-02-12T04:18:59.495 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44413.log 2026-02-12T04:18:59.495 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89241.log.gz 2026-02-12T04:18:59.495 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69377.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-12T04:18:59.495 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69377.log.gz 2026-02-12T04:18:59.496 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32212.log 2026-02-12T04:18:59.496 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44413.log.gz 2026-02-12T04:18:59.496 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31847.log 2026-02-12T04:18:59.496 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.32212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67172.log 2026-02-12T04:18:59.496 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32212.log.gz 2026-02-12T04:18:59.496 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47550.log 2026-02-12T04:18:59.496 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31847.log.gz 2026-02-12T04:18:59.496 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42354.log 2026-02-12T04:18:59.496 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67172.log.gz 2026-02-12T04:18:59.497 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47207.log 2026-02-12T04:18:59.497 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47550.log.gz 2026-02-12T04:18:59.497 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93945.log 2026-02-12T04:18:59.497 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42354.log.gz 2026-02-12T04:18:59.497 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45296.log 2026-02-12T04:18:59.497 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47207.log.gz 2026-02-12T04:18:59.497 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49023.log 2026-02-12T04:18:59.497 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93945.log.gz 2026-02-12T04:18:59.497 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74230.log 2026-02-12T04:18:59.497 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45296.log.gz 2026-02-12T04:18:59.497 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47697.log 2026-02-12T04:18:59.498 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49023.log.gz 2026-02-12T04:18:59.498 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74034.log 2026-02-12T04:18:59.498 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74230.log.gz 2026-02-12T04:18:59.498 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43383.log 2026-02-12T04:18:59.498 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47697.log.gz 2026-02-12T04:18:59.498 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89486.log 2026-02-12T04:18:59.498 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74034.log.gz 2026-02-12T04:18:59.498 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46668.log 2026-02-12T04:18:59.498 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43383.log.gz 2026-02-12T04:18:59.498 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68740.log 2026-02-12T04:18:59.499 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89486.log: /var/log/ceph/ceph-client.admin.46668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40918.log 2026-02-12T04:18:59.499 INFO:teuthology.orchestra.run.trial103.stderr: 28.9% -- replaced with /var/log/ceph/ceph-client.admin.89486.log.gz 2026-02-12T04:18:59.499 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46668.log.gz 2026-02-12T04:18:59.499 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53776.log 2026-02-12T04:18:59.499 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68740.log.gz 2026-02-12T04:18:59.499 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75994.log 2026-02-12T04:18:59.499 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40918.log.gz 2026-02-12T04:18:59.499 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59673.log 2026-02-12T04:18:59.499 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53776.log.gz 2026-02-12T04:18:59.500 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48925.log 2026-02-12T04:18:59.500 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75994.log.gz 2026-02-12T04:18:59.500 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49611.log 2026-02-12T04:18:59.500 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59673.log.gz 2026-02-12T04:18:59.500 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90809.log 2026-02-12T04:18:59.500 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48925.log.gz 2026-02-12T04:18:59.500 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53727.log 2026-02-12T04:18:59.500 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49611.log.gz 2026-02-12T04:18:59.500 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80078.log 2026-02-12T04:18:59.500 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90809.log.gz 2026-02-12T04:18:59.500 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39497.log 2026-02-12T04:18:59.500 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53727.log.gz 2026-02-12T04:18:59.501 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35147.log 2026-02-12T04:18:59.501 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80078.log.gz 2026-02-12T04:18:59.501 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47501.log 2026-02-12T04:18:59.501 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39497.log.gz 2026-02-12T04:18:59.501 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62320.log 2026-02-12T04:18:59.501 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35147.log.gz 2026-02-12T04:18:59.501 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55736.log 2026-02-12T04:18:59.501 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47501.log.gz 2026-02-12T04:18:59.501 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64084.log 2026-02-12T04:18:59.501 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62320.log.gz 2026-02-12T04:18:59.502 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80666.log 2026-02-12T04:18:59.502 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55736.log.gz 2026-02-12T04:18:59.502 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62712.log 2026-02-12T04:18:59.502 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64084.log.gz 2026-02-12T04:18:59.502 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60752.log 2026-02-12T04:18:59.502 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80666.log.gz 2026-02-12T04:18:59.502 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46717.log 2026-02-12T04:18:59.502 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62712.log.gz 2026-02-12T04:18:59.502 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88408.log 2026-02-12T04:18:59.502 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60752.log.gz 2026-02-12T04:18:59.503 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46717.log.gz 2026-02-12T04:18:59.503 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63055.log 2026-02-12T04:18:59.503 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66878.log 2026-02-12T04:18:59.503 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88408.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88408.log.gz 2026-02-12T04:18:59.503 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77807.log 2026-02-12T04:18:59.503 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63055.log.gz 2026-02-12T04:18:59.503 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66878.log.gz 2026-02-12T04:18:59.503 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92132.log 2026-02-12T04:18:59.503 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92328.log 2026-02-12T04:18:59.503 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77807.log.gz 2026-02-12T04:18:59.504 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58839.log 2026-02-12T04:18:59.504 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92132.log.gz 2026-02-12T04:18:59.504 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38369.log 2026-02-12T04:18:59.504 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92328.log.gz 2026-02-12T04:18:59.504 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76974.log 2026-02-12T04:18:59.504 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58839.log.gz 2026-02-12T04:18:59.504 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65310.log 2026-02-12T04:18:59.504 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38369.log.gz 2026-02-12T04:18:59.504 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81548.log 2026-02-12T04:18:59.504 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76974.log.gz 2026-02-12T04:18:59.505 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44609.log 2026-02-12T04:18:59.505 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65310.log.gz 2026-02-12T04:18:59.505 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81548.log.gz 2026-02-12T04:18:59.505 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62957.log 2026-02-12T04:18:59.505 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42648.log 2026-02-12T04:18:59.505 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44609.log.gz 2026-02-12T04:18:59.505 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86252.log 2026-02-12T04:18:59.505 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62957.log.gz 2026-02-12T04:18:59.506 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80715.log 2026-02-12T04:18:59.506 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42648.log.gz 2026-02-12T04:18:59.506 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86252.log.gz 2026-02-12T04:18:59.506 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38271.log 2026-02-12T04:18:59.506 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94876.log 2026-02-12T04:18:59.506 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80715.log.gz 2026-02-12T04:18:59.506 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94533.log 2026-02-12T04:18:59.506 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38271.log.gz 2026-02-12T04:18:59.506 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52796.log 2026-02-12T04:18:59.506 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94876.log.gz 2026-02-12T04:18:59.507 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46129.log 2026-02-12T04:18:59.507 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94533.log.gz 2026-02-12T04:18:59.507 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58250.log 2026-02-12T04:18:59.507 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52796.log.gz 2026-02-12T04:18:59.507 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66829.log 2026-02-12T04:18:59.507 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46129.log.gz 2026-02-12T04:18:59.507 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72711.log 2026-02-12T04:18:59.507 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58250.log.gz 2026-02-12T04:18:59.507 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51032.log 2026-02-12T04:18:59.507 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66829.log.gz 2026-02-12T04:18:59.507 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74671.log 2026-02-12T04:18:59.507 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72711.log.gz 2026-02-12T04:18:59.508 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72074.log 2026-02-12T04:18:59.508 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51032.log.gz 2026-02-12T04:18:59.508 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35611.log 2026-02-12T04:18:59.508 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74671.log.gz 2026-02-12T04:18:59.508 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55785.log 2026-02-12T04:18:59.508 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72074.log.gz 2026-02-12T04:18:59.508 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69722.log 2026-02-12T04:18:59.508 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35611.log.gz 2026-02-12T04:18:59.508 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55785.log.gz 2026-02-12T04:18:59.509 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84586.log 2026-02-12T04:18:59.509 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51277.log 2026-02-12T04:18:59.509 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69722.log.gz 2026-02-12T04:18:59.509 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84586.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.84586.log.gz -- 2026-02-12T04:18:59.509 INFO:teuthology.orchestra.run.trial103.stderr: /var/log/ceph/ceph-client.admin.44070.log 2026-02-12T04:18:59.509 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51277.log.gz 2026-02-12T04:18:59.509 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87330.log 2026-02-12T04:18:59.509 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42550.log 2026-02-12T04:18:59.509 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44070.log.gz 2026-02-12T04:18:59.510 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87330.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.73789.log 2026-02-12T04:18:59.510 INFO:teuthology.orchestra.run.trial103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.87330.log.gz 2026-02-12T04:18:59.510 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47746.log 2026-02-12T04:18:59.510 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42550.log.gz 2026-02-12T04:18:59.510 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75749.log 2026-02-12T04:18:59.510 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73789.log.gz 2026-02-12T04:18:59.510 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40183.log 2026-02-12T04:18:59.510 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47746.log.gz 2026-02-12T04:18:59.510 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85272.log 2026-02-12T04:18:59.510 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75749.log.gz 2026-02-12T04:18:59.511 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71731.log 2026-02-12T04:18:59.511 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40183.log.gz 2026-02-12T04:18:59.511 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54903.log 2026-02-12T04:18:59.511 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85272.log.gz 2026-02-12T04:18:59.511 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93210.log 2026-02-12T04:18:59.511 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71731.log.gz 2026-02-12T04:18:59.511 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76876.log 0.0% 2026-02-12T04:18:59.511 INFO:teuthology.orchestra.run.trial103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54903.log.gz 2026-02-12T04:18:59.511 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70261.log 2026-02-12T04:18:59.511 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93210.log.gz 2026-02-12T04:18:59.512 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82185.log 2026-02-12T04:18:59.512 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76876.log.gz 2026-02-12T04:18:59.512 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50444.log 2026-02-12T04:18:59.512 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70261.log.gz 2026-02-12T04:18:59.512 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32042.log 2026-02-12T04:18:59.512 INFO:teuthology.orchestra.run.trial103.stderr: 29.5% -- replaced with /var/log/ceph/ceph-client.admin.82185.log.gz 2026-02-12T04:18:59.512 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31507.log 2026-02-12T04:18:59.512 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50444.log.gz 2026-02-12T04:18:59.512 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71682.log 2026-02-12T04:18:59.512 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.32042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32042.log.gz 2026-02-12T04:18:59.513 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43972.log 2026-02-12T04:18:59.513 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31507.log.gz 2026-02-12T04:18:59.513 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54609.log 2026-02-12T04:18:59.513 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71682.log.gz 2026-02-12T04:18:59.513 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52257.log 2026-02-12T04:18:59.513 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43972.log.gz 2026-02-12T04:18:59.513 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71584.log 2026-02-12T04:18:59.513 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54609.log.gz 2026-02-12T04:18:59.513 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89339.log 2026-02-12T04:18:59.513 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52257.log.gz 2026-02-12T04:18:59.514 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.71584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41359.log 2026-02-12T04:18:59.514 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71584.log.gz 2026-02-12T04:18:59.514 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95758.log 2026-02-12T04:18:59.514 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89339.log.gz 2026-02-12T04:18:59.514 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86154.log 2026-02-12T04:18:59.514 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41359.log.gz 2026-02-12T04:18:59.514 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94484.log 2026-02-12T04:18:59.514 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95758.log.gz 2026-02-12T04:18:59.514 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77562.log 2026-02-12T04:18:59.514 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86154.log.gz 2026-02-12T04:18:59.515 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66045.log 2026-02-12T04:18:59.515 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94484.log.gz 2026-02-12T04:18:59.515 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40330.log 2026-02-12T04:18:59.515 INFO:teuthology.orchestra.run.trial103.stderr: 29.2% -- replaced with /var/log/ceph/ceph-client.admin.77562.log.gz 2026-02-12T04:18:59.515 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38614.log 2026-02-12T04:18:59.515 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66045.log.gz 2026-02-12T04:18:59.515 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87967.log 2026-02-12T04:18:59.515 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40330.log.gz 2026-02-12T04:18:59.515 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47354.log 2026-02-12T04:18:59.515 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38614.log.gz 2026-02-12T04:18:59.516 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52061.log 2026-02-12T04:18:59.516 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87967.log.gz 2026-02-12T04:18:59.516 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68446.log 2026-02-12T04:18:59.516 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47354.log.gz 2026-02-12T04:18:59.516 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90858.log 2026-02-12T04:18:59.516 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52061.log.gz 2026-02-12T04:18:59.516 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69525.log 2026-02-12T04:18:59.516 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68446.log.gz 2026-02-12T04:18:59.516 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73446.log 2026-02-12T04:18:59.516 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90858.log.gz 2026-02-12T04:18:59.517 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56974.log 2026-02-12T04:18:59.517 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69525.log.gz 2026-02-12T04:18:59.517 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96101.log 2026-02-12T04:18:59.517 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73446.log.gz 2026-02-12T04:18:59.517 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51130.log 2026-02-12T04:18:59.517 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56974.log.gz 2026-02-12T04:18:59.517 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44462.log 2026-02-12T04:18:59.517 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96101.log.gz 2026-02-12T04:18:59.517 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76729.log 2026-02-12T04:18:59.517 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51130.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51130.log.gz 2026-02-12T04:18:59.517 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81303.log 2026-02-12T04:18:59.517 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44462.log.gz 2026-02-12T04:18:59.518 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88653.log 2026-02-12T04:18:59.518 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76729.log.gz 2026-02-12T04:18:59.518 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63937.log 2026-02-12T04:18:59.518 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81303.log.gz 2026-02-12T04:18:59.518 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60997.log 2026-02-12T04:18:59.518 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88653.log.gz 2026-02-12T04:18:59.518 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47648.log 2026-02-12T04:18:59.518 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63937.log.gz 2026-02-12T04:18:59.518 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63545.log 2026-02-12T04:18:59.518 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60997.log.gz 2026-02-12T04:18:59.519 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57562.log 2026-02-12T04:18:59.519 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47648.log.gz 2026-02-12T04:18:59.519 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61242.log 2026-02-12T04:18:59.519 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63545.log.gz 2026-02-12T04:18:59.519 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65849.log 2026-02-12T04:18:59.519 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57562.log.gz 2026-02-12T04:18:59.519 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64918.log 2026-02-12T04:18:59.519 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61242.log.gz 2026-02-12T04:18:59.519 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44560.log 2026-02-12T04:18:59.520 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65849.log.gz 2026-02-12T04:18:59.520 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59574.log 2026-02-12T04:18:59.520 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64918.log.gz 2026-02-12T04:18:59.520 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53972.log 2026-02-12T04:18:59.520 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44560.log.gz 2026-02-12T04:18:59.520 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30623.log 2026-02-12T04:18:59.520 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59574.log.gz 2026-02-12T04:18:59.520 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82822.log 2026-02-12T04:18:59.520 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53972.log.gz 2026-02-12T04:18:59.520 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36996.log 2026-02-12T04:18:59.521 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30623.log.gz 2026-02-12T04:18:59.521 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88163.log 2026-02-12T04:18:59.521 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82822.log.gz 2026-02-12T04:18:59.521 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68642.log 2026-02-12T04:18:59.521 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36996.log.gz 2026-02-12T04:18:59.521 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52845.log 2026-02-12T04:18:59.521 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88163.log.gz 2026-02-12T04:18:59.521 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93847.log 2026-02-12T04:18:59.521 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68642.log.gz 2026-02-12T04:18:59.521 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55246.log 2026-02-12T04:18:59.521 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52845.log.gz 2026-02-12T04:18:59.522 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91593.log 2026-02-12T04:18:59.522 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93847.log.gz 2026-02-12T04:18:59.522 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38516.log 2026-02-12T04:18:59.522 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55246.log.gz 2026-02-12T04:18:59.522 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91446.log 2026-02-12T04:18:59.522 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91593.log.gz 2026-02-12T04:18:59.522 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62271.log 2026-02-12T04:18:59.522 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38516.log.gz 2026-02-12T04:18:59.522 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44217.log 2026-02-12T04:18:59.522 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91446.log.gz 2026-02-12T04:18:59.522 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88898.log 2026-02-12T04:18:59.522 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62271.log.gz 2026-02-12T04:18:59.523 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44952.log 2026-02-12T04:18:59.523 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44217.log.gz 2026-02-12T04:18:59.523 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54462.log 2026-02-12T04:18:59.523 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88898.log.gz 2026-02-12T04:18:59.523 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72172.log 2026-02-12T04:18:59.523 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44952.log.gz 2026-02-12T04:18:59.523 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78261.log 2026-02-12T04:18:59.523 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54462.log.gz 2026-02-12T04:18:59.524 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45541.log 2026-02-12T04:18:59.524 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72172.log.gz 2026-02-12T04:18:59.524 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94043.log 2026-02-12T04:18:59.524 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78261.log.gz 2026-02-12T04:18:59.524 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96738.log 2026-02-12T04:18:59.524 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45541.log.gz 2026-02-12T04:18:59.524 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85125.log 2026-02-12T04:18:59.524 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94043.log.gz 2026-02-12T04:18:59.524 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52159.log 2026-02-12T04:18:59.524 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96738.log.gz 2026-02-12T04:18:59.524 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76141.log 2026-02-12T04:18:59.525 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85125.log.gz 2026-02-12T04:18:59.525 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52159.log.gz 2026-02-12T04:18:59.525 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61634.log 2026-02-12T04:18:59.525 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75259.log 2026-02-12T04:18:59.525 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76141.log.gz 2026-02-12T04:18:59.525 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37290.log 2026-02-12T04:18:59.525 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61634.log.gz 2026-02-12T04:18:59.525 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95709.log 2026-02-12T04:18:59.525 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75259.log.gz 2026-02-12T04:18:59.525 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89633.log 2026-02-12T04:18:59.525 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37290.log.gz 2026-02-12T04:18:59.526 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67270.log 2026-02-12T04:18:59.526 INFO:teuthology.orchestra.run.trial103.stderr: 28.9% -- replaced with /var/log/ceph/ceph-client.admin.95709.log.gz 2026-02-12T04:18:59.526 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77709.log 2026-02-12T04:18:59.526 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89633.log.gz 2026-02-12T04:18:59.526 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56925.log 2026-02-12T04:18:59.526 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67270.log.gz 2026-02-12T04:18:59.526 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81401.log 2026-02-12T04:18:59.526 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77709.log.gz 2026-02-12T04:18:59.526 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67711.log 2026-02-12T04:18:59.526 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56925.log.gz 2026-02-12T04:18:59.526 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35464.log 2026-02-12T04:18:59.526 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81401.log.gz 2026-02-12T04:18:59.527 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73887.log 2026-02-12T04:18:59.527 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67711.log.gz 2026-02-12T04:18:59.527 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36261.log 2026-02-12T04:18:59.527 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35464.log.gz 2026-02-12T04:18:59.527 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63153.log 2026-02-12T04:18:59.527 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73887.log.gz 2026-02-12T04:18:59.527 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76631.log 2026-02-12T04:18:59.527 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36261.log.gz 2026-02-12T04:18:59.527 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64672.log 2026-02-12T04:18:59.527 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63153.log.gz 2026-02-12T04:18:59.528 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77366.log 2026-02-12T04:18:59.528 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76631.log.gz 2026-02-12T04:18:59.528 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36849.log 2026-02-12T04:18:59.528 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64672.log.gz 2026-02-12T04:18:59.528 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56680.log 2026-02-12T04:18:59.528 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77366.log.gz 2026-02-12T04:18:59.528 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36849.log.gz 2026-02-12T04:18:59.528 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91005.log 2026-02-12T04:18:59.528 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56680.log.gz 2026-02-12T04:18:59.528 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86497.log 2026-02-12T04:18:59.529 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63251.log 2026-02-12T04:18:59.529 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91005.log.gz 2026-02-12T04:18:59.529 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90956.log 2026-02-12T04:18:59.529 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86497.log.gz 2026-02-12T04:18:59.529 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50150.log 2026-02-12T04:18:59.529 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63251.log.gz 2026-02-12T04:18:59.529 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79294.log 2026-02-12T04:18:59.529 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90956.log.gz 2026-02-12T04:18:59.530 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40134.log 2026-02-12T04:18:59.530 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50150.log.gz 2026-02-12T04:18:59.530 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94778.log 2026-02-12T04:18:59.530 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79294.log.gz 2026-02-12T04:18:59.530 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57759.log 2026-02-12T04:18:59.530 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40134.log.gz 2026-02-12T04:18:59.530 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89388.log 2026-02-12T04:18:59.530 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94778.log.gz 2026-02-12T04:18:59.530 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62467.log 2026-02-12T04:18:59.530 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57759.log.gz 2026-02-12T04:18:59.531 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46570.log 2026-02-12T04:18:59.531 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89388.log.gz 2026-02-12T04:18:59.531 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41506.log 2026-02-12T04:18:59.531 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62467.log.gz 2026-02-12T04:18:59.531 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43629.log 2026-02-12T04:18:59.531 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46570.log.gz 2026-02-12T04:18:59.531 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68348.log 2026-02-12T04:18:59.531 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41506.log.gz 2026-02-12T04:18:59.531 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78163.log 2026-02-12T04:18:59.531 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43629.log.gz 2026-02-12T04:18:59.531 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62565.log 2026-02-12T04:18:59.531 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68348.log.gz 2026-02-12T04:18:59.532 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96591.log 2026-02-12T04:18:59.532 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78163.log.gz 2026-02-12T04:18:59.532 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61585.log 2026-02-12T04:18:59.532 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62565.log.gz 2026-02-12T04:18:59.532 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57906.log 2026-02-12T04:18:59.532 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96591.log.gz 2026-02-12T04:18:59.532 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37830.log 2026-02-12T04:18:59.532 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61585.log.gz 2026-02-12T04:18:59.532 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82283.log 2026-02-12T04:18:59.532 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57906.log.gz 2026-02-12T04:18:59.532 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.37830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38222.log 2026-02-12T04:18:59.532 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37830.log.gz 2026-02-12T04:18:59.533 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82283.log.gz 2026-02-12T04:18:59.533 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48827.log 2026-02-12T04:18:59.533 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63202.log 2026-02-12T04:18:59.533 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38222.log.gz 2026-02-12T04:18:59.533 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92720.log 2026-02-12T04:18:59.533 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48827.log.gz 2026-02-12T04:18:59.533 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63202.log: 29.5% -- replaced with /var/log/ceph/ceph-client.admin.63202.log.gz 2026-02-12T04:18:59.533 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38026.log 2026-02-12T04:18:59.534 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55099.log 2026-02-12T04:18:59.534 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92720.log.gz 2026-02-12T04:18:59.534 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58888.log 2026-02-12T04:18:59.534 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38026.log.gz 2026-02-12T04:18:59.534 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26008.log 2026-02-12T04:18:59.534 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55099.log.gz 2026-02-12T04:18:59.534 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87526.log 2026-02-12T04:18:59.534 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58888.log.gz 2026-02-12T04:18:59.534 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.26008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87379.log 2026-02-12T04:18:59.534 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26008.log.gz 2026-02-12T04:18:59.535 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74426.log 2026-02-12T04:18:59.535 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87526.log.gz 2026-02-12T04:18:59.535 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52992.log 2026-02-12T04:18:59.535 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87379.log.gz 2026-02-12T04:18:59.535 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46472.log 2026-02-12T04:18:59.535 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74426.log.gz 2026-02-12T04:18:59.535 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91250.log 2026-02-12T04:18:59.535 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52992.log.gz 2026-02-12T04:18:59.535 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79195.log 2026-02-12T04:18:59.535 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46472.log.gz 2026-02-12T04:18:59.535 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91250.log.gz 2026-02-12T04:18:59.536 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95268.log 2026-02-12T04:18:59.536 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.79195.log: /var/log/ceph/ceph-client.admin.79735.log 2026-02-12T04:18:59.536 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79195.log.gz 2026-02-12T04:18:59.536 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95268.log.gz 2026-02-12T04:18:59.536 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48432.log 2026-02-12T04:18:59.536 INFO:teuthology.orchestra.run.trial103.stderr:gzip/var/log/ceph/ceph-client.admin.79735.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.85419.log 2026-02-12T04:18:59.536 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79735.log.gz 2026-02-12T04:18:59.536 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48432.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-12T04:18:59.536 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48432.log.gz 2026-02-12T04:18:59.537 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30624.log 2026-02-12T04:18:59.537 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85419.log.gz 2026-02-12T04:18:59.537 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77072.log 2026-02-12T04:18:59.537 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58643.log 2026-02-12T04:18:59.537 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30624.log.gz 2026-02-12T04:18:59.537 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67025.log 2026-02-12T04:18:59.537 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77072.log.gz 2026-02-12T04:18:59.537 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75504.log 2026-02-12T04:18:59.537 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58643.log.gz 2026-02-12T04:18:59.537 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92671.log 2026-02-12T04:18:59.538 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67025.log.gz 2026-02-12T04:18:59.538 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62418.log 2026-02-12T04:18:59.538 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75504.log.gz 2026-02-12T04:18:59.538 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38957.log 2026-02-12T04:18:59.538 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92671.log.gz 2026-02-12T04:18:59.538 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62418.log.gz 2026-02-12T04:18:59.538 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53041.log 2026-02-12T04:18:59.538 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94680.log 2026-02-12T04:18:59.538 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38957.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38957.log.gz 2026-02-12T04:18:59.538 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95611.log 2026-02-12T04:18:59.539 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53041.log.gz 2026-02-12T04:18:59.539 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76386.log 2026-02-12T04:18:59.539 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94680.log.gz 2026-02-12T04:18:59.539 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84243.log 2026-02-12T04:18:59.539 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95611.log.gz 2026-02-12T04:18:59.539 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85762.log 2026-02-12T04:18:59.539 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76386.log.gz 2026-02-12T04:18:59.539 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44854.log 2026-02-12T04:18:59.539 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84243.log.gz 2026-02-12T04:18:59.539 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86399.log 2026-02-12T04:18:59.539 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85762.log.gz 2026-02-12T04:18:59.540 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44854.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.44854.log.gz -- 2026-02-12T04:18:59.540 INFO:teuthology.orchestra.run.trial103.stderr: /var/log/ceph/ceph-client.admin.43727.log 2026-02-12T04:18:59.540 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64476.log 2026-02-12T04:18:59.540 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86399.log.gz 2026-02-12T04:18:59.540 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88261.log 2026-02-12T04:18:59.540 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43727.log.gz 2026-02-12T04:18:59.540 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69575.log 2026-02-12T04:18:59.540 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64476.log.gz 2026-02-12T04:18:59.540 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35869.log 2026-02-12T04:18:59.540 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88261.log.gz 2026-02-12T04:18:59.541 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67956.log 2026-02-12T04:18:59.541 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69575.log.gz 2026-02-12T04:18:59.541 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34484.log 2026-02-12T04:18:59.541 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35869.log.gz 2026-02-12T04:18:59.541 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86987.log 2026-02-12T04:18:59.541 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67956.log.gz 2026-02-12T04:18:59.541 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.34484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77268.log 2026-02-12T04:18:59.541 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34484.log.gz 2026-02-12T04:18:59.541 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40575.log 2026-02-12T04:18:59.541 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86987.log.gz 2026-02-12T04:18:59.542 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66682.log 2026-02-12T04:18:59.542 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77268.log.gz 2026-02-12T04:18:59.542 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72319.log 2026-02-12T04:18:59.542 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40575.log.gz 2026-02-12T04:18:59.542 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41555.log 2026-02-12T04:18:59.542 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66682.log.gz 2026-02-12T04:18:59.542 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40722.log 2026-02-12T04:18:59.542 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72319.log.gz 2026-02-12T04:18:59.542 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42942.log 2026-02-12T04:18:59.542 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41555.log.gz 2026-02-12T04:18:59.542 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34435.log 2026-02-12T04:18:59.543 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40722.log.gz 2026-02-12T04:18:59.543 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87477.log 2026-02-12T04:18:59.543 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42942.log.gz 2026-02-12T04:18:59.543 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.34435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40820.log 2026-02-12T04:18:59.543 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34435.log.gz 2026-02-12T04:18:59.543 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90319.log 2026-02-12T04:18:59.543 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87477.log.gz 2026-02-12T04:18:59.543 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80127.log 2026-02-12T04:18:59.543 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40820.log.gz 2026-02-12T04:18:59.544 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74279.log 2026-02-12T04:18:59.544 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90319.log.gz 2026-02-12T04:18:59.544 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64623.log 2026-02-12T04:18:59.544 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80127.log.gz 2026-02-12T04:18:59.544 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.74279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73348.log 2026-02-12T04:18:59.544 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74279.log.gz 2026-02-12T04:18:59.544 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88359.log 2026-02-12T04:18:59.544 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64623.log.gz 2026-02-12T04:18:59.544 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70114.log 2026-02-12T04:18:59.544 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73348.log.gz 2026-02-12T04:18:59.544 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88359.log.gz 2026-02-12T04:18:59.544 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35007.log 2026-02-12T04:18:59.545 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73005.log 2026-02-12T04:18:59.545 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70114.log.gz 2026-02-12T04:18:59.545 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94435.log 2026-02-12T04:18:59.545 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35007.log.gz 2026-02-12T04:18:59.545 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57955.log 2026-02-12T04:18:59.545 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73005.log.gz 2026-02-12T04:18:59.545 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46423.log 2026-02-12T04:18:59.545 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94435.log.gz 2026-02-12T04:18:59.546 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76435.log 2026-02-12T04:18:59.546 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57955.log.gz 2026-02-12T04:18:59.546 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50395.log 2026-02-12T04:18:59.546 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46423.log.gz 2026-02-12T04:18:59.546 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94092.log 2026-02-12T04:18:59.546 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76435.log.gz 2026-02-12T04:18:59.546 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51620.log 2026-02-12T04:18:59.546 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50395.log.gz 2026-02-12T04:18:59.546 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46031.log 2026-02-12T04:18:59.546 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94092.log.gz 2026-02-12T04:18:59.546 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51620.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.40036.log -- replaced with /var/log/ceph/ceph-client.admin.51620.log.gz 2026-02-12T04:18:59.546 INFO:teuthology.orchestra.run.trial103.stderr: 2026-02-12T04:18:59.547 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65996.log 2026-02-12T04:18:59.547 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46031.log.gz 2026-02-12T04:18:59.547 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79441.log 2026-02-12T04:18:59.547 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40036.log.gz 2026-02-12T04:18:59.547 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.65996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43825.log 2026-02-12T04:18:59.547 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65996.log.gz 2026-02-12T04:18:59.547 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50640.log 2026-02-12T04:18:59.547 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79441.log.gz 2026-02-12T04:18:59.547 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43825.log.gz 2026-02-12T04:18:59.547 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76925.log 2026-02-12T04:18:59.548 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48285.log 2026-02-12T04:18:59.548 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50640.log.gz 2026-02-12T04:18:59.548 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64035.log 2026-02-12T04:18:59.548 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76925.log.gz 2026-02-12T04:18:59.548 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97032.log 2026-02-12T04:18:59.548 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48285.log.gz 2026-02-12T04:18:59.548 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64035.log: gzip 0.0% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.64035.log.gz /var/log/ceph/ceph-client.admin.26365.log 2026-02-12T04:18:59.548 INFO:teuthology.orchestra.run.trial103.stderr: 2026-02-12T04:18:59.549 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.97032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93700.log 2026-02-12T04:18:59.549 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97032.log.gz 2026-02-12T04:18:59.549 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.26365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75553.log 2026-02-12T04:18:59.549 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26365.log.gz 2026-02-12T04:18:59.549 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93700.log.gz 2026-02-12T04:18:59.549 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66633.log 2026-02-12T04:18:59.549 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62026.log 2026-02-12T04:18:59.549 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75553.log.gz 2026-02-12T04:18:59.550 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60850.log 2026-02-12T04:18:59.550 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66633.log.gz 2026-02-12T04:18:59.550 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75357.log 2026-02-12T04:18:59.550 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62026.log.gz 2026-02-12T04:18:59.550 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56275.log 2026-02-12T04:18:59.550 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60850.log.gz 2026-02-12T04:18:59.550 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75357.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.75357.log.gz 2026-02-12T04:18:59.550 INFO:teuthology.orchestra.run.trial103.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.67858.log 2026-02-12T04:18:59.550 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59329.log 2026-02-12T04:18:59.550 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56275.log.gz 2026-02-12T04:18:59.551 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49513.log 2026-02-12T04:18:59.551 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67858.log.gz 2026-02-12T04:18:59.551 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46619.log 2026-02-12T04:18:59.551 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59329.log.gz 2026-02-12T04:18:59.551 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49513.log.gz 2026-02-12T04:18:59.551 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72662.log 2026-02-12T04:18:59.551 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62173.log 2026-02-12T04:18:59.551 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46619.log.gz 2026-02-12T04:18:59.552 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72760.log 2026-02-12T04:18:59.552 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72662.log.gz 2026-02-12T04:18:59.552 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62173.log.gz 2026-02-12T04:18:59.552 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60458.log 2026-02-12T04:18:59.552 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35148.log 2026-02-12T04:18:59.552 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72760.log.gz 2026-02-12T04:18:59.552 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60458.log.gz 2026-02-12T04:18:59.552 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55344.log 2026-02-12T04:18:59.552 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70996.log 2026-02-12T04:18:59.552 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35148.log.gz 2026-02-12T04:18:59.553 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96199.log 2026-02-12T04:18:59.553 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55344.log.gz 2026-02-12T04:18:59.553 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43040.log 2026-02-12T04:18:59.553 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70996.log.gz 2026-02-12T04:18:59.553 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43923.log 2026-02-12T04:18:59.553 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96199.log.gz 2026-02-12T04:18:59.553 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79392.log 2026-02-12T04:18:59.553 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43040.log.gz 2026-02-12T04:18:59.554 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90760.log 2026-02-12T04:18:59.554 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43923.log.gz 2026-02-12T04:18:59.554 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44511.log 2026-02-12T04:18:59.554 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79392.log.gz 2026-02-12T04:18:59.554 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31913.log 2026-02-12T04:18:59.554 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90760.log.gz 2026-02-12T04:18:59.554 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44511.log.gz 2026-02-12T04:18:59.554 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77660.log 2026-02-12T04:18:59.554 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73593.log 2026-02-12T04:18:59.554 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31913.log.gz 2026-02-12T04:18:59.555 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.77660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78999.log 2026-02-12T04:18:59.555 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77660.log.gz 2026-02-12T04:18:59.555 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.73593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86840.log 2026-02-12T04:18:59.555 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73593.log.gz 2026-02-12T04:18:59.555 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76092.log 2026-02-12T04:18:59.555 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78999.log.gz 2026-02-12T04:18:59.555 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.86840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76239.log 2026-02-12T04:18:59.555 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86840.log.gz 2026-02-12T04:18:59.555 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59133.log 2026-02-12T04:18:59.555 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76092.log.gz 2026-02-12T04:18:59.556 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45933.log 2026-02-12T04:18:59.556 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76239.log.gz 2026-02-12T04:18:59.556 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79588.log 2026-02-12T04:18:59.556 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59133.log.gz 2026-02-12T04:18:59.556 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.45933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57366.log 2026-02-12T04:18:59.556 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45933.log.gz 2026-02-12T04:18:59.556 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79588.log.gz 2026-02-12T04:18:59.556 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95660.log 2026-02-12T04:18:59.556 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93112.log 2026-02-12T04:18:59.556 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57366.log.gz 2026-02-12T04:18:59.557 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43285.log 2026-02-12T04:18:59.557 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95660.log.gz 2026-02-12T04:18:59.557 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54168.log 2026-02-12T04:18:59.557 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93112.log.gz 2026-02-12T04:18:59.557 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.43285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96787.log 2026-02-12T04:18:59.557 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43285.log.gz 2026-02-12T04:18:59.557 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54168.log.gz 2026-02-12T04:18:59.557 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-12T04:18:59.557 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52649.log 2026-02-12T04:18:59.558 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96787.log.gz 2026-02-12T04:18:59.558 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51424.log 2026-02-12T04:18:59.558 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31846.log 2026-02-12T04:18:59.558 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52649.log.gz 2026-02-12T04:18:59.558 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81107.log 2026-02-12T04:18:59.558 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51424.log.gz 2026-02-12T04:18:59.558 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.31846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93651.log 2026-02-12T04:18:59.558 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31846.log.gz 2026-02-12T04:18:59.559 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51228.log 2026-02-12T04:18:59.559 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81107.log.gz 2026-02-12T04:18:59.559 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78212.log 2026-02-12T04:18:59.559 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93651.log.gz 2026-02-12T04:18:59.559 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59771.log 2026-02-12T04:18:59.559 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51228.log.gz 2026-02-12T04:18:59.559 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62908.log 2026-02-12T04:18:59.559 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78212.log.gz 2026-02-12T04:18:59.559 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41702.log 2026-02-12T04:18:59.559 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59771.log.gz 2026-02-12T04:18:59.560 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48775.log 2026-02-12T04:18:59.560 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62908.log.gz 2026-02-12T04:18:59.560 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36653.log 2026-02-12T04:18:59.560 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41702.log.gz 2026-02-12T04:18:59.560 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48334.log 2026-02-12T04:18:59.560 INFO:teuthology.orchestra.run.trial103.stderr: 28.5% -- replaced with /var/log/ceph/ceph-client.admin.48775.log.gz 2026-02-12T04:18:59.560 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96346.log 2026-02-12T04:18:59.560 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36653.log.gz 2026-02-12T04:18:59.560 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32100.log 2026-02-12T04:18:59.560 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48334.log.gz 2026-02-12T04:18:59.560 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41065.log 2026-02-12T04:18:59.561 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96346.log.gz 2026-02-12T04:18:59.561 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.32100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83410.log 2026-02-12T04:18:59.561 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32100.log.gz 2026-02-12T04:18:59.561 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53139.log 2026-02-12T04:18:59.561 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41065.log.gz 2026-02-12T04:18:59.561 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48138.log 2026-02-12T04:18:59.561 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83410.log.gz 2026-02-12T04:18:59.561 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39350.log 2026-02-12T04:18:59.561 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53139.log.gz 2026-02-12T04:18:59.561 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.48138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44364.log 2026-02-12T04:18:59.561 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48138.log.gz 2026-02-12T04:18:59.562 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59476.log 2026-02-12T04:18:59.562 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39350.log.gz 2026-02-12T04:18:59.562 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.44364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68544.log 2026-02-12T04:18:59.562 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44364.log.gz 2026-02-12T04:18:59.562 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68936.log 2026-02-12T04:18:59.562 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59476.log.gz 2026-02-12T04:18:59.562 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41947.log 2026-02-12T04:18:59.562 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68544.log.gz 2026-02-12T04:18:59.563 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38761.log 2026-02-12T04:18:59.563 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68936.log.gz 2026-02-12T04:18:59.563 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96150.log 2026-02-12T04:18:59.563 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41947.log.gz 2026-02-12T04:18:59.563 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94141.log 2026-02-12T04:18:59.563 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38761.log.gz 2026-02-12T04:18:59.563 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.96150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85174.log 2026-02-12T04:18:59.563 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96150.log.gz 2026-02-12T04:18:59.563 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94141.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.54952.log -- replaced with /var/log/ceph/ceph-client.admin.94141.log.gz 2026-02-12T04:18:59.563 INFO:teuthology.orchestra.run.trial103.stderr: 2026-02-12T04:18:59.563 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84537.log 2026-02-12T04:18:59.563 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85174.log.gz 2026-02-12T04:18:59.564 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.54952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55883.log 2026-02-12T04:18:59.564 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54952.log.gz 2026-02-12T04:18:59.564 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35709.log 2026-02-12T04:18:59.564 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84537.log.gz 2026-02-12T04:18:59.564 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38565.log 2026-02-12T04:18:59.564 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55883.log.gz 2026-02-12T04:18:59.564 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35709.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.35709.log.gz -- 2026-02-12T04:18:59.564 INFO:teuthology.orchestra.run.trial103.stderr: /var/log/ceph/ceph-client.admin.53433.log 2026-02-12T04:18:59.564 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36800.log 2026-02-12T04:18:59.564 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38565.log.gz 2026-02-12T04:18:59.565 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68201.log 2026-02-12T04:18:59.565 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53433.log.gz 2026-02-12T04:18:59.565 INFO:teuthology.orchestra.run.trial103.stderr: 95.5% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-12T04:18:59.565 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.36800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76582.log 2026-02-12T04:18:59.565 INFO:teuthology.orchestra.run.trial103.stderr: 28.9% -- replaced with /var/log/ceph/ceph-client.admin.36800.log.gz 2026-02-12T04:18:59.565 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.68201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76533.log 2026-02-12T04:18:59.565 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68201.log.gz 2026-02-12T04:18:59.565 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89437.log 2026-02-12T04:18:59.565 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76582.log.gz 2026-02-12T04:18:59.565 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35205.log 2026-02-12T04:18:59.565 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76533.log.gz 2026-02-12T04:18:59.566 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89437.log.gz 2026-02-12T04:18:59.566 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62761.log 2026-02-12T04:18:59.566 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.35205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80813.log 2026-02-12T04:18:59.566 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35205.log.gz 2026-02-12T04:18:59.566 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.62761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69279.log 2026-02-12T04:18:59.566 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62761.log.gz 2026-02-12T04:18:59.566 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72809.log 2026-02-12T04:18:59.566 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80813.log.gz 2026-02-12T04:18:59.566 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57857.log 2026-02-12T04:18:59.567 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69279.log.gz 2026-02-12T04:18:59.567 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93749.log 2026-02-12T04:18:59.567 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72809.log.gz 2026-02-12T04:18:59.567 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83312.log 2026-02-12T04:18:59.567 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57857.log.gz 2026-02-12T04:18:59.567 INFO:teuthology.orchestra.run.trial103.stderr:gzip/var/log/ceph/ceph-client.admin.93749.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.55932.log 2026-02-12T04:18:59.567 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93749.log.gz 2026-02-12T04:18:59.567 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39399.log 2026-02-12T04:18:59.567 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83312.log.gz 2026-02-12T04:18:59.567 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55932.log.gz 2026-02-12T04:18:59.568 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80372.log 2026-02-12T04:18:59.568 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.39399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47256.log 2026-02-12T04:18:59.568 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39399.log.gz 2026-02-12T04:18:59.568 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58398.log 2026-02-12T04:18:59.568 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80372.log.gz 2026-02-12T04:18:59.568 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72858.log 2026-02-12T04:18:59.568 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47256.log.gz 2026-02-12T04:18:59.569 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63006.log 2026-02-12T04:18:59.569 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58398.log.gz 2026-02-12T04:18:59.569 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49807.log 2026-02-12T04:18:59.569 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72858.log.gz 2026-02-12T04:18:59.569 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.63006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53531.log 2026-02-12T04:18:59.569 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63006.log.gz 2026-02-12T04:18:59.569 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.49807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49807.log.gz 2026-02-12T04:18:59.569 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51767.log 2026-02-12T04:18:59.569 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.53531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56373.log 2026-02-12T04:18:59.569 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53531.log.gz 2026-02-12T04:18:59.570 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.51767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67123.log 2026-02-12T04:18:59.570 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51767.log.gz 2026-02-12T04:18:59.570 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56373.log.gzgzip 2026-02-12T04:18:59.570 INFO:teuthology.orchestra.run.trial103.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38859.log 2026-02-12T04:18:59.570 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.67123.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67123.log.gz 2026-02-12T04:18:59.570 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72025.log 2026-02-12T04:18:59.570 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66241.log 2026-02-12T04:18:59.570 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38859.log.gz 2026-02-12T04:18:59.571 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75014.log 2026-02-12T04:18:59.571 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72025.log.gz 2026-02-12T04:18:59.571 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58447.log 2026-02-12T04:18:59.571 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66241.log.gz 2026-02-12T04:18:59.571 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56128.log 2026-02-12T04:18:59.571 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75014.log.gz 2026-02-12T04:18:59.571 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.58447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75847.log 2026-02-12T04:18:59.571 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58447.log.gz 2026-02-12T04:18:59.572 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.56128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81009.log 2026-02-12T04:18:59.572 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56128.log.gz 2026-02-12T04:18:59.572 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50591.log 2026-02-12T04:18:59.572 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75847.log.gz 2026-02-12T04:18:59.572 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.81009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26725.log 2026-02-12T04:18:59.572 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81009.log.gz 2026-02-12T04:18:59.572 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.50591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41408.log 2026-02-12T04:18:59.572 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50591.log.gz 2026-02-12T04:18:59.572 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.26725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33231.log 2026-02-12T04:18:59.572 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26725.log.gz 2026-02-12T04:18:59.572 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.41408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95121.log 2026-02-12T04:18:59.572 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41408.log.gz 2026-02-12T04:18:59.573 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.33231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46962.log 2026-02-12T04:18:59.573 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33231.log.gz 2026-02-12T04:18:59.573 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64280.log 2026-02-12T04:18:59.573 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95121.log.gz 2026-02-12T04:18:59.573 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.46962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60556.log 2026-02-12T04:18:59.573 INFO:teuthology.orchestra.run.trial103.stderr: 29.8% -- replaced with /var/log/ceph/ceph-client.admin.46962.log.gz 2026-02-12T04:18:59.573 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60311.log 2026-02-12T04:18:59.573 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64280.log.gz 2026-02-12T04:18:59.573 INFO:teuthology.orchestra.run.trial103.stderr:gzip/var/log/ceph/ceph-client.admin.60556.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.69181.log 2026-02-12T04:18:59.573 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60556.log.gz 2026-02-12T04:18:59.574 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90466.log 2026-02-12T04:18:59.574 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60311.log.gz 2026-02-12T04:18:59.574 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61389.log 2026-02-12T04:18:59.574 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69181.log.gz 2026-02-12T04:18:59.574 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52306.log 2026-02-12T04:18:59.574 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90466.log.gz 2026-02-12T04:18:59.574 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64967.log 2026-02-12T04:18:59.574 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61389.log.gz 2026-02-12T04:18:59.574 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70163.log 2026-02-12T04:18:59.574 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52306.log.gz 2026-02-12T04:18:59.575 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78114.log 2026-02-12T04:18:59.575 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64967.log.gz 2026-02-12T04:18:59.575 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.70163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76043.log 2026-02-12T04:18:59.575 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70163.log.gz 2026-02-12T04:18:59.575 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42991.log 2026-02-12T04:18:59.575 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78114.log.gz 2026-02-12T04:18:59.575 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.76043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40232.log 2026-02-12T04:18:59.575 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76043.log.gz 2026-02-12T04:18:59.575 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.42991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64771.log 2026-02-12T04:18:59.575 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42991.log.gz 2026-02-12T04:18:59.575 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66731.log 2026-02-12T04:18:59.575 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40232.log.gz 2026-02-12T04:18:59.576 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79048.log 2026-02-12T04:18:59.576 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64771.log.gz 2026-02-12T04:18:59.576 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90711.log 2026-02-12T04:18:59.576 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66731.log.gz 2026-02-12T04:18:59.576 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47795.log 2026-02-12T04:18:59.576 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79048.log.gz 2026-02-12T04:18:59.576 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.90711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66437.log 2026-02-12T04:18:59.576 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90711.log.gz 2026-02-12T04:18:59.576 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79784.log 2026-02-12T04:18:59.576 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47795.log.gz 2026-02-12T04:18:59.577 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80911.log 2026-02-12T04:18:59.577 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66437.log.gz 2026-02-12T04:18:59.577 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84390.log 2026-02-12T04:18:59.577 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79784.log.gz 2026-02-12T04:18:59.577 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52110.log 2026-02-12T04:18:59.577 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80911.log.gz 2026-02-12T04:18:59.577 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30825.log 2026-02-12T04:18:59.577 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84390.log.gz 2026-02-12T04:18:59.577 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79097.log 2026-02-12T04:18:59.577 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52110.log.gz 2026-02-12T04:18:59.578 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80274.log 2026-02-12T04:18:59.578 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30825.log.gz 2026-02-12T04:18:59.578 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61536.log 2026-02-12T04:18:59.578 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79097.log.gz 2026-02-12T04:18:59.578 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.80274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78555.log 2026-02-12T04:18:59.578 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80274.log.gz 2026-02-12T04:18:59.578 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.61536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75945.log 2026-02-12T04:18:59.578 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61536.log.gz 2026-02-12T04:18:59.578 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55834.log 2026-02-12T04:18:59.578 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78555.log.gz 2026-02-12T04:18:59.579 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.75945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30428.log 2026-02-12T04:18:59.579 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75945.log.gz 2026-02-12T04:18:59.579 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.55834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93504.log 2026-02-12T04:18:59.579 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55834.log.gz 2026-02-12T04:18:59.579 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34386.log 2026-02-12T04:18:59.579 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30428.log.gz 2026-02-12T04:18:59.579 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.93504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79686.log 2026-02-12T04:18:59.579 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93504.log.gz 2026-02-12T04:18:59.579 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.34386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85615.log 2026-02-12T04:18:59.579 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34386.log.gz 2026-02-12T04:18:59.580 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95170.log 2026-02-12T04:18:59.580 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79686.log.gz 2026-02-12T04:18:59.580 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.85615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95856.log 2026-02-12T04:18:59.580 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85615.log.gz 2026-02-12T04:18:59.580 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94386.log 2026-02-12T04:18:59.580 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95170.log.gz 2026-02-12T04:18:59.580 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.95856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89094.log 2026-02-12T04:18:59.580 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95856.log.gz 2026-02-12T04:18:59.580 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.94386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66976.log 2026-02-12T04:18:59.580 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94386.log.gz 2026-02-12T04:18:59.581 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.89094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66094.log 2026-02-12T04:18:59.581 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89094.log.gz 2026-02-12T04:18:59.581 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78506.log 2026-02-12T04:18:59.581 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66976.log.gz 2026-02-12T04:18:59.581 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.66094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79245.log 2026-02-12T04:18:59.581 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66094.log.gz 2026-02-12T04:18:59.581 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92034.log 2026-02-12T04:18:59.581 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78506.log.gz 2026-02-12T04:18:59.581 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.79245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88212.log 2026-02-12T04:18:59.581 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79245.log.gz 2026-02-12T04:18:59.581 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.92034.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.92034.log.gz 2026-02-12T04:18:59.582 INFO:teuthology.orchestra.run.trial103.stderr: /var/log/ceph/ceph-client.admin.82724.log 2026-02-12T04:18:59.582 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.88212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60507.log 2026-02-12T04:18:59.582 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88212.log.gz 2026-02-12T04:18:59.582 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.82724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60115.log 2026-02-12T04:18:59.582 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82724.log.gz 2026-02-12T04:18:59.582 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52698.log 2026-02-12T04:18:59.582 INFO:teuthology.orchestra.run.trial103.stderr: 56.8% -- replaced with /var/log/ceph/ceph-client.admin.60507.log.gz 2026-02-12T04:18:59.582 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.60115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84439.log 2026-02-12T04:18:59.582 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60115.log.gz 2026-02-12T04:18:59.582 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.52698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78359.log 2026-02-12T04:18:59.582 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52698.log.gz 2026-02-12T04:18:59.583 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.84439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84439.log.gz 2026-02-12T04:18:59.583 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69034.log 2026-02-12T04:18:59.583 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40428.log 2026-02-12T04:18:59.583 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78359.log.gz 2026-02-12T04:18:59.583 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.69034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47893.log 2026-02-12T04:18:59.583 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69034.log.gz 2026-02-12T04:18:59.583 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87134.log 2026-02-12T04:18:59.583 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40428.log.gz 2026-02-12T04:18:59.584 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.47893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30425.log 2026-02-12T04:18:59.584 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47893.log.gz 2026-02-12T04:18:59.584 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38467.log 2026-02-12T04:18:59.584 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87134.log.gz 2026-02-12T04:18:59.584 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59182.log 2026-02-12T04:18:59.584 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30425.log.gz 2026-02-12T04:18:59.584 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.38467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91936.log 2026-02-12T04:18:59.584 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38467.log.gz 2026-02-12T04:18:59.584 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.59182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40771.log 2026-02-12T04:18:59.584 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59182.log.gz 2026-02-12T04:18:59.584 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.91936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72907.log 2026-02-12T04:18:59.584 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91936.log.gz 2026-02-12T04:18:59.585 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.40771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64427.log 2026-02-12T04:18:59.585 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40771.log.gz 2026-02-12T04:18:59.585 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.72907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57661.log 2026-02-12T04:18:59.585 INFO:teuthology.orchestra.run.trial103.stderr: 29.2% -- replaced with /var/log/ceph/ceph-client.admin.72907.log.gz 2026-02-12T04:18:59.585 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.64427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78310.log 2026-02-12T04:18:59.585 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64427.log.gz 2026-02-12T04:18:59.585 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.57661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83361.log 2026-02-12T04:18:59.585 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57661.log.gz 2026-02-12T04:18:59.585 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.78310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30622.log 2026-02-12T04:18:59.585 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78310.log.gz 2026-02-12T04:18:59.586 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.83361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87624.log 2026-02-12T04:18:59.586 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83361.log.gz 2026-02-12T04:18:59.586 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.30622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30622.log.gz 2026-02-12T04:18:59.586 INFO:teuthology.orchestra.run.trial103.stderr:/var/log/ceph/ceph-client.admin.87624.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87624.log.gz 2026-02-12T04:18:59.644 INFO:teuthology.orchestra.run.trial103.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2026-02-12T04:18:59.680 INFO:teuthology.orchestra.run.trial103.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-12T04:18:59.772 INFO:teuthology.orchestra.run.trial103.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2026-02-12T04:18:59.849 INFO:teuthology.orchestra.run.trial103.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-12T04:19:13.665 INFO:teuthology.orchestra.run.trial103.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-12T04:19:23.939 INFO:teuthology.orchestra.run.trial103.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-12T04:19:50.130 INFO:teuthology.orchestra.run.trial103.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2026-02-12T04:19:50.131 INFO:teuthology.orchestra.run.trial103.stderr: 2026-02-12T04:19:50.131 INFO:teuthology.orchestra.run.trial103.stderr:real 0m50.841s 2026-02-12T04:19:50.131 INFO:teuthology.orchestra.run.trial103.stderr:user 1m27.383s 2026-02-12T04:19:50.131 INFO:teuthology.orchestra.run.trial103.stderr:sys 0m4.125s 2026-02-12T04:19:50.132 INFO:tasks.ceph:Archiving logs... 2026-02-12T04:19:50.132 DEBUG:teuthology.misc:Transferring archived files from trial103:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-11_20:27:55-rados-wip-yuri3-testing-2026-02-09-1550-tentacle-distro-default-trial/45778/remote/trial103/log 2026-02-12T04:19:50.133 DEBUG:teuthology.orchestra.run.trial103:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-12T04:19:57.888 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f66ecfb34c0370410e78b3ee641753d19da653b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_1279775fef08a9169052c6dd6a10581353958aac/qa/tasks/ceph.py", line 2011, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_1279775fef08a9169052c6dd6a10581353958aac/qa/tasks/ceph_manager.py", line 2930, in wait_for_clean assert time.time() - start < timeout, \ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: wait_for_clean: failed before timeout expired 2026-02-12T04:19:57.889 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-12T04:19:57.910 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f66ecfb34c0370410e78b3ee641753d19da653b/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5f66ecfb34c0370410e78b3ee641753d19da653b/teuthology/task/install/__init__.py", line 644, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f66ecfb34c0370410e78b3ee641753d19da653b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_1279775fef08a9169052c6dd6a10581353958aac/qa/tasks/ceph.py", line 2011, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_1279775fef08a9169052c6dd6a10581353958aac/qa/tasks/ceph_manager.py", line 2930, in wait_for_clean assert time.time() - start < timeout, \ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: wait_for_clean: failed before timeout expired 2026-02-12T04:19:57.911 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-12T04:19:57.911 DEBUG:teuthology.orchestra.run.trial103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-12T04:19:57.993 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-12T04:19:58.004 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-12T04:19:58.005 DEBUG:teuthology.orchestra.run.trial103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-12T04:19:58.021 INFO:teuthology.orchestra.run.trial103.stderr:bash: line 1: ntpq: command not found 2026-02-12T04:19:58.036 INFO:teuthology.orchestra.run.trial103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-12T04:19:58.036 INFO:teuthology.orchestra.run.trial103.stdout:=============================================================================== 2026-02-12T04:19:58.036 INFO:teuthology.orchestra.run.trial103.stdout:^- 170-187-147-56.ip.linode> 2 6 377 51 -5474us[-5474us] +/- 79ms 2026-02-12T04:19:58.036 INFO:teuthology.orchestra.run.trial103.stdout:^- us-central-1.clearnet.pw 2 7 377 47 +3562us[+3562us] +/- 85ms 2026-02-12T04:19:58.036 INFO:teuthology.orchestra.run.trial103.stdout:^* time.cloudflare.com 3 7 377 51 +81us[ +108us] +/- 6948us 2026-02-12T04:19:58.036 INFO:teuthology.orchestra.run.trial103.stdout:^- time1.weber.edu 1 6 377 50 +596us[ +596us] +/- 27ms 2026-02-12T04:19:58.036 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-12T04:19:58.047 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-12T04:19:58.047 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-12T04:19:58.061 DEBUG:teuthology.orchestra.run.trial103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-12T04:19:58.136 DEBUG:teuthology.orchestra.run.trial103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-12T04:19:58.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T04:19:58.161 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-12T04:19:58.174 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-12T04:19:58.188 INFO:teuthology.task.internal:Duration was 1678.186126 seconds 2026-02-12T04:19:58.188 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-12T04:19:58.202 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-12T04:19:58.202 DEBUG:teuthology.orchestra.run.trial103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-12T04:19:58.237 INFO:teuthology.orchestra.run.trial103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-12T04:19:58.493 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-12T04:19:58.493 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial103.front.sepia.ceph.com 2026-02-12T04:19:58.493 DEBUG:teuthology.orchestra.run.trial103:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-12T04:19:58.551 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-12T04:19:58.551 DEBUG:teuthology.orchestra.run.trial103:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-12T04:19:58.652 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-12T04:19:58.652 DEBUG:teuthology.orchestra.run.trial103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-12T04:19:58.674 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T04:19:58.674 INFO:teuthology.orchestra.run.trial103.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T04:19:58.674 INFO:teuthology.orchestra.run.trial103.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-12T04:19:58.674 INFO:teuthology.orchestra.run.trial103.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-12T04:19:58.675 INFO:teuthology.orchestra.run.trial103.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-12T04:19:58.681 INFO:teuthology.orchestra.run.trial103.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 93.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-12T04:19:58.683 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-12T04:19:58.693 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-12T04:19:58.693 DEBUG:teuthology.orchestra.run.trial103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-12T04:19:58.747 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-12T04:19:58.759 DEBUG:teuthology.orchestra.run.trial103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-12T04:19:58.810 INFO:teuthology.orchestra.run.trial103.stdout:kernel.core_pattern = core 2026-02-12T04:19:58.824 DEBUG:teuthology.orchestra.run.trial103:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-12T04:19:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T04:19:58.880 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-12T04:19:58.891 INFO:teuthology.task.internal:Transferring archived files... 2026-02-12T04:19:58.892 DEBUG:teuthology.misc:Transferring archived files from trial103:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-11_20:27:55-rados-wip-yuri3-testing-2026-02-09-1550-tentacle-distro-default-trial/45778/remote/trial103 2026-02-12T04:19:58.892 DEBUG:teuthology.orchestra.run.trial103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-12T04:19:58.950 INFO:teuthology.task.internal:Removing archive directory... 2026-02-12T04:19:58.950 DEBUG:teuthology.orchestra.run.trial103:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-12T04:19:59.006 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-12T04:19:59.019 INFO:teuthology.task.internal:Not uploading archives. 2026-02-12T04:19:59.019 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-12T04:19:59.032 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-12T04:19:59.032 DEBUG:teuthology.orchestra.run.trial103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-12T04:19:59.062 INFO:teuthology.orchestra.run.trial103.stdout: 786525 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 04:19 /home/ubuntu/cephtest 2026-02-12T04:19:59.063 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-12T04:19:59.076 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-12T04:19:59.092 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: AssertionError: wait_for_clean: failed before timeout expired 2026-02-12T04:19:59.093 INFO:teuthology.run:Summary data: description: rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/many msgr/async-v2only objectstore/{bluestore/{alloc$/{btree} base mem$/{low} onode-segment$/{1M} write$/{v2/{compr$/{no$/{no}} v2}}}} rados supported-random-distro$/{centos_latest}} duration: 1678.18612575531 failure_reason: '' flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com sentry_event: null status: fail success: false 2026-02-12T04:19:59.093 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-12T04:19:59.175 INFO:teuthology.run:FAIL